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ABSTRACT 



This evaluation of the information dissemination program of 
the National Center for Research on Evaluation, Standards, and Student 
Testing (CRESST) uses a inclusive approach to gather, analyze, and report 
information to help decision makers. Results of three questionnaires, reviews 
and analyses of CRESST products, and Internet records were used to determine 
what is known about CRESST research and its products, how its research and 
products are used, and the perceived quality and usefulness of CRESST 
research and products overall and across user groups. Responses of 875 
educators (43% of the total sample) were received for the CRESST descriptive 
questionnaire. Major findings are: (1) across 18 items measuring the quality 

of CRESST research and development, 73 to 91% of consumers rated CRESST in 
the top 3 categories of performance on an 8-point scale; (2) few significant 
differences exist between CRESST consumers, so that market segmentation of 
research does not seem appropriate; (3) the CRESST web site has more than 
tripled the number of CRESST products available; (4) impact from CRESST 
research is considerable, but changes in instruments could document that 
impact better; (5) comparative information from other research organizations 
would aid decision makers in measuring program quality; and (6) future CRESST 
dissemination planning should develop strategies for evaluation of major new 
CRESST programs. Five appendixes contain the questionnaires, descriptive 
statistics, and normality plots and descriptions of nonparametric tests. 
(Contains 22 tables, 22 figures, and 99 references.) (Author/SLD) 
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Since the first national education research centers were established in the mid- 
1960s, there has been frequent pressure, usually at a national level, to transfer the 
centers’ best research findings into practice to improve teaching, learning, and student 
achievement. Lack of evidence that center research is reaching schools and having an 
impact has led to frequent criticism of center dissemination strategies. Unfortunately, 
evaluations of centers’ dissemination methods for program improvement are scarce as 
are customer ratings of center research quality and usefulness. This evaluation of the 
dissemination program at the National Center for Research on Evaluation, Standards, 
and Student Testing (CRESST) uses an inclusive approach to gather, analyze, and 
report dependable and useful information to center decision makers. Three newly 
designed fast-back questionnaires combined with a comprehensive review and analysis 




ERIC 



of center product and Internet records were used to answer the following evaluation 
questions: 1) what do we know about the existing use of CRESST research and its 
products; 2) how is CRESST research used and shared by research consumers and what 
impact results from its use; 3) what is the perceived quality and usefulness of CRESST 
research and dissemination; 4) what differences exist in perceived quality and usefulness 
across different consumer groups and different types of registration lists; and finally, 5) 
what else was learned about the quality and usefulness of CRESST research that would 
be helpful to program decision makers? The major findings of this study were: 1) 
across 18 items measuring the quality of CRESST research and development, 73% - 
91% of consumers rated CRESST in the top three categories of performance on an 
eight-point scale; 2) few significant differences exist between CRESST consumers or 
between three different types of CRESST registration lists, thus market segmentation of 
products and research does not seem appropriate; 3) the CRESST web site has more 
than tripled the number of products now reaching consumers; 4) impact from CRESST 
research is considerable, but changes in instruments could better document that impact; 
5) com-parative information from other research centers or similar organizations would 
aid decision makers in measuring program quality, usefulness, impact, and needed areas 
of improvement; and 6) future CRESST dissemination planning should develop 
strategies for evaluation of major new CRESST programs, including the 
CRESST/LAUSD collaboration (Assessment and Instmction Models), Quality Schools 
Portfolios, and the Quality Education Forum. 




xiv 



16 



Chapter 1 

INTRODUCTION TO THE PROBLEM 



Please keep me on [the] CRESST Line [mailing list] and make research 
available to those of us in the field. Political decisions arc being made and we 



need the data. 



5th/6th grade California teacher 



ERIC 



Backgrounds 

Directors and conununication directors at federally funded education research 
centers confront a complex enviromnent There is high demand for the research 
information and products they produce as suggested by the above quote from a 
frustrated California teacher. Her voice is shared by many. A director of state testing 
writes to CRESST that he needs more research on the impact of standards-based reform. 
A teacher reports that she has given the CRESST’s assessment research information to 
her principal to read, but she sees no result on practice. A senior project director for the 
Los Angeles County Department of Education says she needs better access to CRESST 
researchers to give presentations across the county. Finally, a science teacher at the 
Florida School for the Deaf and Blind suggests that a useful CRESST product would be 
computer software for the management of grades, portfolios, and smdent self-evaluation 
programs. He adds that “ease of use is an essential component since teacher time is 
earmarked for many other tasks." Meanwhile, an official from the U.S. Department of 
Education needs documentation of impact from CRESST and other research centers to 
demonstrate to Congress that education research is making a difference in schools. 
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The HftmanHs for education research are diverse and overwhelming, but limited 
resources challenge center leadership to focus research and dissemination where it can 
fnalcft the greatest difference. Especially challenging are expectations for what 
dissemination can do that exceed researchers capabilities to deliver (Weiss, 1989). 

Policy makers become frustrated because research and its dissemination do not seem to 
have a strong enough impact on schools and student performance (Cross, 1990, 1991; 
Kaestle, 1993). Researchers, usually reliant on policy makers for funding, attempt to 
increase dissemination, but have very limited budgets with which to reach the large 
audiences envisioned (Price, 1984). Dissemination suffers, especially when it comes at 
the end of a rather long research cycle when funding may not be renewed and research 
staff must be let go. A self-deprecating cycle thus ensues with policy makers reluctant 
to increase research budgets because research, and its disse m ination, hasn’t made a 
difference. Never mind that the education center prevented the waste of millions of 
dollars because its research showed that a practice or a test should “not” be 
implemented. Or that its research has had transparent effect through hundreds of change 
agents who transfer research strands into practice every day and into almost every 
school. Research will make a difference when test scores go up. Tension between 
policy makers and researchers remains high. 

Research and knowledge utilization; on definitions and paradigms. 

Given the multitude of factors involved in educational reform and improvement, 
how do we know when research makes a difference? Little agreement exists, for 
example, on even a basic definition or paradigm of knowledge utilization in the 
classroom. Knowledge utilization may be very broad: “Knowledge utilization occurs 
when an exchange of purposely prepared information is communicated to a set of 
recipients” (Louis, et al., 1984). Or it may be much more complex such as Backer’s 



(1991; 1993) eight categories of knowledge utilization: technology transfer (hard and 
soft); information dissemination and utilization, research utilization, innovative 
diffusion, sociology of knowledge, organizational change, policy research, 
interpersonal and mass communication. Knowledge in the form of consumer 
information (Komoski, 1989) used for decision making purposes is another variant. A 
complicating factor is that overly exists between knowledge utilization categories, i.e., 
an Internet Web site is both a type of mass communication, a form of information 
dissemination, and possibly consumer information. Obviously the more specific one 
becomes in defining knowledge utilization or research use of any type, the more often a 
use will fall into multiple categories. 

Dissemination definitions and categories are similarly slippery: one classification 
scheme (Buttram, Rosenblum, & Brigham 1992) uses just two categories: passive, 
knowledge transfer through quiescent means, such as publications; and proactive, 
knowledge transfer through operative methods, usually including direct human 
interaction, such as a workshop. The Dissemination and Analysis Group (DAG) 
promoted the definition of dissemination based on usage (Hutchins, 1989): 

• Usage 1 : Spread — the one-way casting out of knowledge in all its forms: 
Information, products, ideas and materials, “as though sowing seeds.” 

• Usage 2: Exchange — The two-way or multi-way flow of information, 
products, ideas, and materials as to needs, problems, and potential solutions. 

• Usage 3: Choice — The facilitation of rational consideration and selection 
among those ideas, materials, outcomes research and development, effective 
educational practices and other knowledge that can be used for the 
improvement of education. 
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• Usage 4: Inq)lementation — technical assistance, training or interpersonal 
activities to increase the use of knowledge or R&D to change attitudes or 
behavior.' 

Finally, Paul Hood (1991) classified the various knowledge utilization concepts 
into the following two paradigms: 

1 ) The Dissemination Paradigm, in which some form of knowledge, produced 
someplace, is broadly disseminated to many users, often at some distance, 
physically and sometimes culturally, from the point of knowledge production. 
Thus the knowledge is "external" to the user system. From the knowledge 
producer's standpoint, the challenge is to communicate with and achieve 
producer-intended forms of use of this knowledge among many potential 
users. Dissemination, marketing, mass media, on-line information systems, 
800 numbers for information services become some of the vehicles for 
co mm unication between producer and user. 

2) The Systemic Change Process Paradigm, in which the main focus of 
knowledge use and production is in one location-whether that is a person's 
head or a large organization. The knowledge use process is local, complex 
and dynamic. And most of the knowledge production is "local." Externally 
produced knowledge (ideas, products, programs, technologies) may be 
stimulative or facilitative, but this use of external knowledge is often 
incidental or subordinate to achieving synergistic changes in group or 
organizational structures, policies, operating procedures, and perhaps even 



‘ Klein and Gwaltney (1990, April) add that most federal dissemination programs use more than one of 
these dissemination methods and that they are not linear or hierarchical. 
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the woiking environment of the organization and in the attitudes, skills, 
motives, values, and shared visions of those involved in this form of systemic 
change process. "Research-based" knowledge may be represented more by a 
process of local disciplined inquiry and reflection than by the products of 
externally produced research. But this is rarely an either-or situation, rather it 
is a melding of knowledge in many forms from many sources. 



What should become q)parent frx>m these various definitions and two paradigms 
is the fragmentation that exists within the knowledge and research utilization fields, 
thereby confounding the dile mma of knowing when research is making a difference. 

Research user as consumer^ 

Paradigm’s notwithstanding, the need to evaluate and therefore somehow 
measure the impact of research, continues. Because our focus in this evaluation is 
primarily in Paul Hood’s Dissemination paradigm, Komoski’s (1989) evaluation 
approach from a customer or consumer’s perspective has appeal. His focus on 
educational product use, mostly related to curriculum materials, might be applied to the 
use of educational research and its various dissemination products, including 
newsletters, videotapes. Internet-based information, CD ROMs, presentations, etc. 
Similar to recent business research promoting the idea of “relationship marketing’’ in 
consumer markets (Christy, Oliver, & Perm, 1996) educational research use shares 
similar characteristics including: 

• customers need for information borne out of uncertainty and conflicting 
information; 

• general psychological attractions to long term relationships; 




5 



21 



ability to pay a premium price if the product or service warrants it; 






Other relationship marketing characteristics that seem to be gaining appeal in 
educational settings include: 

• the potential for a high degree of customization to the consumer’s needs; 

• the customer’s perceived need for training, not unlike a new piece of software; 

• customer involvement in the design process. 



Evaluation of such dissemination products and impact could adapt Scriven’s 
consumer-based evaluation theory that focuses on consumers and their needs (1973a). 
Such an t^proach lends itself well to marketing research and a consumer-driven 
measurement methodology as suggested by Brown (1996): 

• customers segmented according to their needs which are determined at least once 
a year, 

• large samples of customers surveyed twice a year and a large percentage (50 
percent or more) of the surveys are answered; 

• customer satisfaction telephone and mail surveys evaluated and continually 
improved; 

• hard data such as repeat business collected to supplement data on customers’ 
opinions of the organization’s products/services. 
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This dissertation, an evaluation of the dissemination program of the National 
Center for Research on Evaluation, Standards, and Student Testing (CRESST), follows 
along Brown’s methodology. Funded since 1985 by the Office of Educational Research 
and Improvement, a branch of the U.S. Department of Education, CRESST conducts 
five-year research programs on national, state, and local issues of educational quality, 
addressing persistent problems in the design and use of assessment systems to serve 
multiple purposes. UCLA is the lead CRESST institution and has partners at the 
University of Colorado, at Boulder, Stanford University; RAND Corporation; the 
University of Pittsburgh; the Educational Testing Service; and the University of 
California, Santa Barbara. Co-directed by two of the leading K-12 educational 
assessment experts in the nation, UCLA Professor Eva L. Baker and University of 
Colorado Professor Robert L. Linn, CRESST has an established presence as an 
educational research center and maintains an active dissemination program. While in 
general we discuss the research centers and their research partners as the same entity, it 
may in fact be more accurate to give prominence to the research parmers, in nearly all 
instances the director or co-directors, who form the collaborations and prepare the grant 
proposals which win funding for the center. Each of these is a well-known person or 
persons in their specific educational research field who typically represent the top 
research expertise in the country. 

Evaluation Objectives. 

CRESST uses a multitude of dissemination methods to communicate its findings 
and theories to a diverse community of education policy makers; state, county, district, 
and local educators; and the public. Presentations, conferences, meetings, and phone 
calls, supplement a widespread print, media, and technology-based communications 
program. The primary purpose of this evaluation was to collect and produce useful data 
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that would lead to important decisions about alternative courses of action (Cronbach, 
1962; Alkin, 1969; 1990a; 1990b) for the CRESST dissemination program. Another 
goal was to develop and suggest methodology and data that will eventually help answer 
the question, “is research making a difference?” Formative in nature, this evaluation is 
integrated into the five-year CRESST research plan (1996-2001). 
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CHAPTER 2: 



LITERATURE REVIEW 

Purposes for Evaluating an Education R&D Disse mination Program 

There are many purposes for evaluation of programs that have application to 
dissemination programs at national education R&D centers. Improving the management 
of program decisions is an oft-mentioned primary purpose for evaluation (Byrk & Light, 
1981) and is closely related to several evaluation definitions (AUcin, 1969; Alkin & 
House, 1992; Cronbach, 1962; Patton, 1982; Stufflebeam, 1973) that emphasize data 
gathering for improved decision making. Generating periodic checks on the 
effectiveness of education programs and determining at which point improvements are 
necessary (Madaus & StufQebeam, 1989) is another evaluation purpose as is the need to 
monitor expenditure of public funds (Alkin & Solmon, 1983; Worthen and Sanders, 
1987) or for resource allocation purposes. Less often stated is the need to evaluate 
because: a) program requirements dictate an evaluation be conducted, or b) to respond to 
historical pressures to improve programs in some specific areas. It is to this very last 
purpose (b) that we first turn, because the pressure to improve education R&D 
dissemination has been long-standing and continues to exert great pressure on both 
researchers and disseminators. 

Historical Need To Improve Education ResearchLJQissemination Programs 

Early years. Pressure to improve the dissemination of education research at 
the federally funded education research centers is not new. The Campbell Report 
(1975), for example, criticized dissemination efforts of the National Institute of 





Education (NIE) and its associated research centers and regional educational laboratories 
for reasons that may sound as familiar today as they did 20 years ago.^ 

We understand the political pressure for “dissemination” of the results of R&D, 
but we conclude that NIE has done little to attack the problem as a substantive 
matter or cluster of issues and competing conceptualizations. We do not think 
that work in the field can be halted until theory catches up, but we do believe an 
experimental attitude would be helpful even as action goes forward, and that 
diverse groups within NIE could be brought together more directly to consider 
paradigms for change and the various roles of “dissemination” within them. 
Research on knowledge-utilization could be more extensively funded as an 
essential basis for policy in this area. 

The Campbell report expressed the belief that the centers and labs were suffering 
from “growing pains,” while a second major report just four years later. Research and 
Development Centers and Regional Educational Laboratories: Strengthening and 
Stabilizing A National Resource saw these organizations as “maturing” (Salmon-Cox, 
1981). The maturing centers and labs were still perceived as needing improved 
dissemination, especially in creating better links to practice, a theme that was to become 
commonplace in future years. 

We see too little attention to forms of dissemination that are firmly linked to the 
inpovement of practice and too little integration among the efforts that 



^National education research centers were first established under the Cooperative Research Act in 1963-64 
while the first regional laboratories were authorized under the Elementary and Secondary Act in 1965-1966. 
Twenty regional labs and 14 research centers were funded in the mid-1960’s under institutional grants or 
contracts. (Source: Center for Leadership Development, Los Angeles, CA 1984) 
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exist... NIE should develop a comprehensive policy on its role in dissemination, 
should conduct programs that are consistent with that policy, and should 
implement effective procedures for the dissemination of the results of the R&D it 



During the same year an article in Educational Researcher (Sharp & Frankel, 
1979) concluded that education research dissemination was the area of lowest emphasis 
for virtually all education research institutions except for state agencies and large public 
school systems. The inference was that university-based research placed little emphasis 
on getting research into the hands of practitioners.^ 

Not that centers or the educational laboratories were ignoring dissemination. In 
fact, there were more programs in the 1970s than at any other time in recent history 
attempting to link education lesearch-to-practice or in a number of cases conducting 
research on dissemination itself. A few of the specific programs related to improved 
dissemination included the Educational Dissemination Studies Program (EDSP), the 
Research and Development Exchange (RDx), and the Regional Services Program 
(RSP). 

Suggesting that research on dissemination itself was a vital activity, EDSP 
contained a special studies component to examine or conduct exploratory studies in 
dissemination. A second component of EDSP, Dissemination and Utilization Studies, 
was responsible for collecting, analyzing, and communicating information supporting 



^ By 1989 the ERIC system had produced more than 6,000 documents related to knowledge or information 
use with at least a substantial amount dedicated to transferring research to practice. Source: Hood, 1989. 



supports. 



Panel for the Review of Laboratory and Center Operations, 1979 
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research dissemination and utilization, and the transmission of knowledge to 
practitioners (Hood, Cates, & McKibbin, 1980). 

RDx had a research-to-practice focus emphasizing the “feed-forward” concept of 
R&D results to practice; coordination of dissemination/school improvement programs; 
information, assistance, and training to schools; and using client and response needs to 
affect future R&D programs. Almost the entire program existed within seven regional 
laboratories with one support service at a university center. RDx reflected the critical role 
that practitioners could play in the “production, synthesis and delivery of new 
knowledge” (Salmon-Cox, 1981). 

The RSP program was quite diverse in its function, but was also housed at 
laboratories with the primary purpose to apply “short-term problems identified by the 
clients in the region served” (Lallmang, 1980). 

Yet another program at about the same time was the Research and Development 
Utilization Program which provided schools with the type of in-person assistance 
advocated in much of the research-to-practice literature. This program assisted schools 
in identifying problem areas that could be improved and then matching the R&D 
resources necessary to implement change (Crandall, 1989). Two other programs worth 
noting were the State Dissemination Capacity Building Program and the Local Problem 
Solving Program. Both supported research-to-practice but with different methods. 

There was also a Dissemination and Resources Group at NIE during this period, 
providing further evidence that the government has had a fairly long history of 
at temptin g to transfer education research- to-practice.^ 



* Several other programs worth mentioning are the National Diffusion Network (NDN) that was 
established in 1974 and funded until 1996. NDN uses state facilitators to directly assist schools in 
defining needs and assisting them with the implementation of educational programs that have been certified 
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Mid and late 1980s. Many of these programs were not to make it through the 
1980s, not because there was less interest or need for dissemination, but at least 
substantially because of attacks on the federal budget. Even the Nation at Risk report in 
1983 did not bring the resources necessary to create dissemination programs capable of 
creating lasting school reform (Crandall, 1989). 

Countering the prevailing perception of failed federal R&D dissemination 
programs was a comprehensive ten-volume report series from Crandall & Associates 
(1983) that documented the various dissemination strategies during the previous 25 
years. Their conclusion was that the previous dissemination efforts across many 
separate but cooperating organizations had successfully produced change (Hutchins, 
1989). Nevertheless, a generally held negative opinion of federal R&D dissemination 
did not disappear. 

A key report in the tnid-80s once again heightened concerns about 
dissemination. Creating and Disseminating Knowledge for Educational Reform: Policy 
Management of the National Institute of Education ’s Regional Educational Laboratories 
and National Research and Development Centers (Price, 1984) revisited the 
effectiveness of existing dissemination programs at centers and labs. Based on the 
study results, dissemination was less than impressive. When a sample of California 
county and local school superintendents (N=93) were surveyed and asked how 
effectively centers and laboratories disseminated research, 41% reported that they “did 
not know,” 34% said “ineffective” and only 25% said “effective.” When the 



by the Department of Education’s Program Effectiveness Panel. Another program sponsored by the Office 
of Education in the early 1970s was the Project Information Packages. Each package consisted of a 
detailed materials kit that could be implemented without on-site assistance. The program had only limited 
success. (Source: Crandall, 1989). 
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superintendents were asked if they had utilized any center or lab research, 64% said no 
or don’t know, while 36% said that they utilized some research or research products.* 

In another part of the study, 72 national education state policy makers and R&D 
specialists were interviewed about their perceptions of the effectiveness of centers and 
labs. When asked how successful the labs had been in accomplishing their objectives, 
44% said “successful,” 28% said “did not know” and 28% said “unsuccessful.” Asked 
the same question of the research centers, two thirds (64%) said that they were 
“successful,” 17% said “don’t know” and 19% saw them as “unsuccessful.”* 

NBE didn’t fare well either. Asked if NIE was successful in accomplishing its 
goals, 52% of the state policy makers and R&D specialists said “successful,” 11% 
responded “don’t know,” and 37% said “unsuccessfiil.” When questioned about the 
impact of R&D work from NIE, centers, and labs on local school districts, 64% said 
“some or no impact,” 7% “didn’t know,” and 29% said “much or a great deal of 
impact.” When asked about the impact of the National Council on Educational Research 
(NCER) on NIE’s labs and centers, 55% answered “some or no impact,” 27% said 
“didn’t know,” and only 18% said “much impact.” Considering that national state 
policy makers and R&D providers should have been a prime audience for federal 
education research, the authors of the study concluded: 

In summary, these responses by knowledgeable, nationally recognized 
e d ii ratinnal Statesmen and R&D specialists do not represent a strong 
endorsement of the dissemination results of N(TER, NIE, or its Labs or Centers. 



^ The argument could be made that these results were indeed not as bad as they seem considering that 
small dissemination budgets were trying to reach large audiences. 

‘ A gain , these results could be inteipreted as somewhat favorable when compared to other programs. 
Nevertheless, the statements by the writers of the report suggest a negative inteipretation. 
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FinaUy, this same study reported results from NIE’s own 1 1 public meetings 
across the United States prior to an upcoming recompetition of centers and labs. The 
results of oral and written testimony from 458 education stakeholders was not favorable 
about center and lab dissemination. The report found that; 

“dissemination activities of labs and centers were criticized by all special interests 
who testified. They [persons who testified] recommended that stakeholders be 
part of labs and centers governing bodies, which should be changed to cany out 
policy-making, instead of advisory functions, at all stages of the R&D cycle.” 

The concern about dissemination led to one of four major conclusions in the 

report 



“NIE's lab and center research has not been effectively delivered and shared with 
local school teachers, administrators, policy-makers, and parents.” 

While critical of the overall center and lab inq)act and dissemination, the report 
acknowledged the huge mission facing the entire education R&D community, 
suggesting that dissemination might target narrow segments of high R&D users. 

Effective dissemination of R&D to potential stakeholders in our nation’s 
decentralized system of education (over 15,500 local school districts and 32(X) 
colleges and universities alone) represents an overwhelming challenge to the 
relatively small resource cq)acity of NIE’s 17 Labs and Centers. This striking 
resource imbalance argues that even greater priority attention be given to 
targeting and disseminating strategies and more careful management of scarce 
federal resources if R&D is to have a cost effective infract on improving 
educational practices. 
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An article in Educational Researcher, The Awful Reputation of Education 

I 

Research (Kaestle, 1993) pointed to the historical dilemma in trying to disseminate 
research findings, exemplified by the comments of a former OERI assistant secretary in 
the late 1980s. 

. . .it isn’t easy to reach the classroom. Chester Firm, former head of OERI and 
the chief author of the famous What Works pamphlet of 1986, questions the 
pamphlet’s efficacy, even though it was widely heralded and a half million 
copies were distributed. Its 41 findings, Finn says, “were validated by quite a lot 
of research, peer reviewed, signed off on by senior people in the field... and 
written in English.’’ It got “maximum White House hoopla.’’ A year later. Firm 
had lunch with 18 high school principals in San Diego. He held up a copy of 
What Works and asked how many had seen it and used it. Four had heard of it, 
two had seen it, and one of those two had discussed it at a faculty meeting. “The 
conclusion I draw from this,’’ says Firm, “is that the print and the dissemination 
media, no matter how skillfully done, won’t work. Therefore, dissemination of 
educational research, if it’s to be done, has to be done some other way... I’m 
almost in despair on this subject, absolutely stymied.’’ 

The late 1980s also saw a changing role for research dissemination at the 
centers. Because the original linear concept of labs utilizing and disseminating center 
research findings appeared to be ineffective, centers came under pressure to increase 
their own dissemination programs. The Subcommittee on Select Education of the 
Committee on Education and Labor (Owens, 1988) attempted to reclaim the original 
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vision of the federal government’s responsibility for education research.^ Their report, 
Educational Research, Development, and Dissemination: Reclaiming a Vision of the 
Federal Role for the 1990’ s and Beyond, was critical of center and lab programs, 
sounding a familiar note: 

What emerged from the extensive oral and written testimony of 21 witnesses 
was a picture of federal educational research and development in disarray. 

The report criticized OERI for not meeting the perceived needs of the schools 
with its current research agenda and dissemination policies. One of the major 
reconunendations was to implement formal evaluation of all federal research programs, 
including dissemination: 

OERI must require routine professional and independent evaluations of all 
funded research, development and dissemination activities and make them 
available to Congress in the form of a biaimual report. 

The 1990s. The Owens’ report also caUed for an independent panel to develop 
a set of evaluation criteria to review education research including centers and labs. But 
funding was never provided for either the panel or for outside evaluations. Labs and 
centers instead were permitted to evaluate their own effectiveness in the next major 
recompetition of 1990. 

It was also in 1990 that Assistant Secretary Christopher Cross, Office of 
Educational Research and Improvement, made it clear that dissemination of education 



^ Not everyone was in complete agreement that dissemination of education research needed increased 
emphasis at this time. In 1987, the Government Accounting Office questioned the emphasis on 
dissemination vs. the production of information (Datta, 1989). 
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research was a top priority. Cross published a brief policy paper in April, 1990, An 
Education Dissemination Policy. 

Our dissemination goal is to ensure that the information needed to support 
education improvement is understandable, accessible, timely, relevant, and 
useful. We will better understand and meet the needs of those for whom our 
information and resources hold promise. Based on this understanding, we will 
provide the best information that can be constructively used to meet these needs. 
This is our responsibility to the American taxpayer, to those who make and 
im plftmftnt public policy, to the professionals who must educate our children and 
to their parents. . .The nature of the problem dictates that we put this policy into 
effect immediately. Each day that we delay, the nation loses. 

The recompetition of centers in 1990 resulted in stronger dissemination 
programs for centers and labs, but improving dissemination was still a concern. Cross 
(1991) emphasized dissemination of federal education research in a speech before the 
annual meeting of the American Association of Colleges for Teacher Educatron. 

I am equally critical perhaps more so, of the research community for failing to 
put key findings in clear language and get it into the hands of teachers in a timely 
fashion. Dissemination has been one of my top priorities. Inside my agency, we 
arc going to great lengths to listen to what teachers say about the research and 
practice inf o rmati on they need and to supply it in a way they find usable. And 
we have translated this dissemination commitment into formal requirements for 
our research centers and our 10 regional laboratories. 

FmphasiTing the OERI Commitment to research dissemination at this time were 
p lans to create a research center dedicated to dissemination and the use of knowledge. 
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the Center for Research on Dissemination and Knowledge Utilization (Klein & 
Gwaltney, 1990). The center was never funded (Louis, 1992). 

Nevertheless, dissemination of research continued to be an area of great concern, 
but specific direction and increased funding remained lacking. Research centers were 
largely allowed to develop and in^)lement their own dissemination programs with little 
guidance from OERI and with the ERIC clearinghouses as the primary method of 
disse minating research at any centralized level.^ 

The need for better dissemination of education research had not escaped the ears 
of the research community. New Models for Disseminating Education R&D, a special 
issue of the journal. Knowledge: Creation. Diffusion, Utilization, (LaFollette, 1992) 
featured articles by many well-known researchers in the education dissemination field. 
Each author published their viewpoints and recommendations for improved 
dissemination methods.’ One general emphasis seemed to be that more systematic 



' Even here, there are multiple ERIC Clearinghouses based on topics which must compete for funding 
every five years. 

’Articles included: Foreword to Education 200S: The Role of Research and Development in an 
Overwhelming Campaign for Education in America, Major R. Owens; The Leadership Role of the U.S. 
Department of Education in Creating and Supporting a National Education Dissemination System, 
Matthew B. Miles and Charles F. Haughey; Prospectus for a Cooperative Extension System in Education, 
Everett M. Rogers; A Framework for Redesigning an R&D-Based National Education Dissemination 
System in the United States, Susan Shurberg Klein; Comparative Perspectives on Dissemination and 
Knowledge Use Policies: Supporting School Improvement, Karen Seashore Louis; Disseminating 
Research Information to Multiple Stakeholders: Lessons From the Experience of the National Institute on 
Disability and Rehabilitation Research, Ellen Liberti Blasiotti; The Distinct Education Extension Agent as 
“Strategic Brokei”: Toward a New Vision for Federally Sponsored Dissemination, Laurence Peters; Tilling 
Fertile Soil: Principles to Guide Transplants From Agriculture to Education Dissemination, Susan 
Shurberg Klein; Excerpts From In the National Interest: The Federal Government in the Reform of K-12 
Math and Science Education, Carnegie Commission on Science, Technology, and Government; 
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dissemination across many organizations would lead to improved dissemination. Many 
of the reconunendations would be addressed in the upcoming OERI reauthorization 
legislation (S. S. Klein, personal conununication, June, 1995) and at least some of them 
would be reflected in the 1995 RFP for national education centers (“Application for a 
grant,” 1995). 

In 1992, a new report emerged that defined what would eventually become an 
integral part of the Goals 2000 legislation. Research and Education Reforms: Roles for 
the Office of Educational Research and Improvement, (Atkinson & Jackson, 1992) 
recommended the reorganization of OERI into directorates, recognizing that past 
reorganizations have had little effect. Dissemination would be more centralized under a 
new office, the Reform Assistance Directorate (RAD) providing reform assistance to 
parents, schools, districts, states. Congress, professional commercial publishers, and 
employers. RAD would assume responsibility for a multitude of programs including 
regional laboratories, OERI publications, electronic networking, and ERIC. Thus, RAD 
would be a central point for disseminating research to a wide variety of stakeholders 
with a renewed emphasis on research-to-practice. 

The Goals 2000 Educate America Ac/ (1994) mirrored much of the 1992 
Atkinson/Jackson report by reflecting great emphasis on dissemination. The chapter of 
the legislation that reauthorized OERI, Title IX— Educational Research and 
Improvement, mentions dissemination no fewer than 72 times. The legislation created a 
new OERI department called the Office of Reform Assistance and Dissemination 
(ORAD) with duties very similar to that suggested for RAD. Even though ORAD is not 
responsible for the research centers, it is charged with coordinating dissemination of all 
directorates. 



Commentaiy: Toward an Unrestricted Dissemination of Research Results, Robert Perloff. 
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Summary of historical context. It becomes obvious that over the last thirty 
years, the majority of reports and studies addressing dissemination of education R&D 
have found dissemination programs lacking and have pressured the research community 
to improve. Researchers and disseminators continue to struggle against an overall 
impression that the dissemination system has failed. It is likely that pressures to 
improve will continue based on the Government Performance and Results Act (1993) 
that requires federal government agencies to set specific goals and develop indicators for 
achieving program results. As with all areas of the federal government, the impact on 
education R&D dissemination will be considerable in that the Act requires performance 
plans, goals, and measurable objectives by the fiscal year 1999. Each year thereafter, 
federal agencies must submit an aimual report that reviews the success of achieving 
performance goals and plans. 

Additional Context for Evaluating an R&D Center Dissemination Program 
Regardless of the multiple purposes or historical pressures to evaluate 
dissemination programs, few published evaluations, either internal or external, are 
available of dissemination programs, especially with a focus on product effectiveness or 
impact. While there are irmumerable studies on research utilization, these are nearly all 
studies related to the systemic change agent paradigm. In fact, only a few published 
studies have attempted to measure the usefulness of the types of products produced by 
most research centers, such as technical reports, newsletters, video programs, 
conference presentations by center staff, etc. Some centers and laboratories have used 
product information surveys enclosed in newsletters, feedback cards, or internal quality 
assurance systems.'** But the first two of these do not represent random samples and 



'"Based on material received after a request to approximately 30 lab and center communicators. 



thus are likely to produce biased results while the latter has a very different, albeit, 
important purpose, that is, to monitor the quality of research, not dissemination. 

There are some tracer studies of education products from Research for Better 
Schools (Buttram, Rosenblum, & Brigham, 1992) and at WestEd" (Mills & Stephens, 
1990; Mills, Tyler, Hood, & Barfield, 1992) that may be useful. In particular, the 
studies by Stephen Mills at WestEd are quite informative based on their comprehensive 
evaluation of printed materials created by the California Department of Education. But 
the material disseminated was largely to consumers who needed information at precise 
moments in time and who required implementation types of support, therefore, not 
necessarily a center research audience. Tracer studies were also conducted by House 
(1996) focusing on impact from several major CRESST authored publications. 

Various research criteria and possible research designs related to dissemination 
exist, including those established for the Program Effectiveness Panel (Cook, Dwyer, & 
Stalford, 1991; Ralph & Dwyer, 1988) where the intention was to document the positive 
effects of irmovative large-scale education programs. However, these criteria and many 
other dissemination research designs generally have relevance to very specific education 
curriculum programs meant for direct classroom implementation. Thus they are more 
appropriate to evaluating the implementation of research information to end-users such 
as practitioners than for the dissemination of knowledge development itself (Miles & 
Haughey, 1992). 

The most useful study was referred by Susan Klein, a dissemination and 
evaluation specialist at the Office of Educational Research and Improvement. The report 
was from the National Center for Education Statistics (Stalford & Stems, 1990) Use of 



" WestEd was formerly the Far West Laboratory for Education Research and Development 



Educational Research and Development Resources by Public Schools Districts. This 
study consisted of a fast-back survey sent to a sample of public school districts across 
the United States concerning their receipt and use of research and development resources 
produced by labs and centers. While the survey is limited to only one type of audience, 
school districts, it served as a starting point for instruments that will be discussed in 
greater detail under the research design section of this dissertation. 

Why do so few evaluations of center (and laboratory) research dissemination 
programs exist, especially given tremendous interest in improving dissemination? It 
could be due to a variety of reasons including organizational factors, lack of resources, 
program diversity, lack of experience for those who might typically be expected to 
conduct such an evaluation, shortage of previous methodology that is time and cost- 
efficient, or any influence created by system interaction of any of the above. 

Barriers to Evaluating Center Dissemination Proerams” 

Or ganiza tional Factors. Evaluation is a component of each OERI education 
R&D center’s mandate and part of the OERI center’s con^tition (Applications for 
grants, 1990; “Application for a grant,” 1995). But evaluation of center research 
programs has not always been effective. A Review of the NIE Evaluation of Research 
and Development Centers and Educational Laboratories (Alkin, 1973) questioned the 
objectivity of the evaluation review panel personnel, implementation of the evaluation, 
outcomes, and criteria. More recently, Diane Ravitch, a former assistant secretary of 
OERI concluded that evaluation of the centers and laboratories was a futile experience 
(1995). Additionally, evaluation practices vary widely between centers and use of the 



'^e baiiiers HiisriiRsed are based on four sources of information: a general literature review, two OERI 
center communicator meetings, general discussions with other center communicators, and personal insight 
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results by OERI for decision making appear to be minimal.'^ Although centers regularly 
report the results of their dissemination projects via quarterly reports to OERI, these 
internally developed documents usually provide updates of individual dissemination 
projects, rough estimates of the numbers and types of audiences reached, and anticipated 
next quarter activities. Other types of reports are occasionally required such as a 
layman’s report, an impact, or accomplishments report. However, use of the results by 
OERI has been small in recent years.'^ The result of these combined effects is that 
evaluation may not be perceived as a necessary or useful tool. 

Further, although not always the case, evaluation tends to occur towards the end 
of a center’s funding cycle. By that time, the results are unlikely to inform stakeholders, 
especially in those cases where a center’s funding will be discontinued. Instability in the 
research centers themselves has been common since the funding of the first centers in 
the mid-1960s. Either new centers have been created, changed as a result of 
recompetition, or simply unfunded and thereby, eliminated. Fewer than half of the 25 
centers funded in 1992 exist in early-1997. Centers set to terminate may have little 
motivation to evaluate their research or dissemination efforts. 

It is also possible that in the final year of a center’s funding OERI may be in the 
midst of another reorganization or the OERI Assistant Secretary recently appointed or 
soon to be appointed with a new slant on dissemination. Thus centers may take a “wait 
and let’s see’’ approach as opposed to using evaluation to influence their own decisions 
towards improved dissemination. Finally, the fifth year of a project is usually a time to 



as a center communicator. 

Based on a informal memo to many center disseminators in March, 1995 and previous use of material 
provided to OERI. 

'* From the same informal memo, it does appear that the Center for Research on Educational 
Accountability and Teacher Evaluation may be an exception. 
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look ahead towards future research needs or write a new proposal. The cumulative 
effect from the above discussion is that evaluation of dissemination does not appear to 
be a well integrated process either by OERI or across research centers. 

Resource shortage. Another major barrier to evaluation of center 
dissemination programs is the same ailment that plagues nearly all education research 
programs: lack of human and/or financial resources. In the 1970s, for example, 
agricultural funding for dissemination in the cooperative extension service was 47 cents 
of each R&D dollar. Meanwhile only 10 cents of every dollar was being spent for 
dissemination of education R&D (Raizen, 1979). Even in the early 1990s funding of 
some R&D centers has been so low that some principal dissemination staff have had 
duties well outside their dissemination area, thereby cutting down their time devoted to 
dissemination, much less evaluation of dissemination programs. The current OERI 
legislation provides minimum funding of future centers at no less than $1.5 milli on per 
year, but even this amount may be inadequate to launch an effective evaluation of 
dissemination programs considering that dissemination and evaluation share small slices 
of a center’s R&D program. 

Program Diversity. Even if staffing was adequate and a more stringent 
evaluation required of dissemination programs, another barrier to effective evaluations is 
represented by the diversity of current dissemination programs supported by centers. 
For example, the Center for Research on the Education of Students Placed At Risk 
(CRESPAR) disseminates a specific education reform program. Success For All, to 
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schools across the United States.'® Approved as a National Diffusion Network program, 
this is a systemic change process dissemination program lacking at most other centers. 
Such programs take years to develop and many centers may not view dissemination of 
complete learning programs as part of their research mission.'® Instead most centers use 
more passive methods such as formal conference presentations and various media. For 
the latter, dissemination methods and emphases vary significantly. The National 
Research Center on the Gifted and Talented (University of Connecticut), for example, 
produces an annual satellite broadcast while other centers such as the National Center on 
Adult Literacy (University of Pennsylvania) and the National Research Center on 
Student Learning (University of Pittsburgh) devote significant resources to electronic 
dissemination. Some centers have placed importance on creating publications specific to 
their audiences, such as the annual Bibliography on School Restructuring produced by 
the former National Center on the Study of Organization and Restructuring Schools 
(University of Wisconsin). Several centers exhibit at major education conferences such 
as AERA. This diversity does not necessarily lend itself to evaluation methods that are 
easily transferred across centers. 

Other barriers. While many of the forgoing reasons are barriers easily 
hurdled by an experienced evaluator, most center dissemination evaluations are likely to 
be conducted by center communicators with little, if any, evaluation expertise. They are 
not likely to have sophisticated knowledge of instrument design or understand the 
importance of instrument or test validity. An additional barrier is that many center 



The Center is also involved in a large number of other dissemination methods. 

“This is not to say that we might not draw on research-to-practice studies such as the Dissemination 
Eftbrts Supporting School Improvement (DESSI) study by Crandall & Associates, 1983 or research done 
by Paul Hood and others at WestEd. 




26 



42 



co mmunic ators are relatively new to the field of education, or if they are experienced 
educators, may have limited communications experience. 

Finally, centers have only sporadically shared the results of their dissemination 
prog rams and thus little is known about what works and what doesn’t” Evaluation of 
dissemination has barely been touched. Systematic methods or instruments supporting 
evaluation of typical center dissemination programs are lacking. 

Brief Review of Marketing Research Literature 

Marketing and the education community. As various for-profit 
corporations begin to manage schools or school districts, such as the Edison Schools 
Project or Education Alternatives Inc., integration of business and K-12 education 
comes closer to reality, albeit still on a small scale. Sophisticated maiketing or 
advertising methods have come with them, such as Channel One, a predecessor of 
Edison Schools and a marketing strategy whereby schools receive free televisions and 
videotape equipment in exchange for a requirement that students view ten minutes of 
Channel One programming daily, complete with commercial advertising. Sophisticated 
marketing techniques have come to the research community also (Walsh, 1996). For 
example, researchers from the New Standards Project have used hand-held electronic 
polling devices to measure community responses to draft benchmaiks for student 
learning, facilitating the evaluation of educational reform ideas on various audiences 
before they are implemented. The case is put forth here that there may be other areas of 
marketing research where the educational research use field might benefit frx)m 
maiketing research, especially as it relates to consumer satisfaction studies. 



'^Many, but not all disseminators from the OERI-funded research centers met in 1993 and 1994. Only a 
small number attended a 1995 AERA communicators meeting. 
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Consumerism is not new to education. As mentioned earlier, criteria for the 
evaluation of educational products (Klein, 1976) and for educational programs as part 
of the National Diffusion Network (Ralph & Dwyer, 1988) have been long-established, 
and Blaine Worthen and James Sanders devote an entire chapter to consumer-oriented 
evaluation approaches in Educational Evaluation: Alternative Approaches and Practical 
Guidelines (1987). The Educational Products Information Exchange (EPIE) Institute 
operated for over twenty years as a consumers’ union for education products and 
services, largely evaluating the value of curriculum-based materials (EPIE Institute, 
1977). Scriven (1974) developed a comprehensive criteria for evaluating any educational 
product. But research centers don’t oftentimes produce curriculum-based research 
products and therefore indicators for such products hold little relevancy for most center 
dissemination programs. 

On the other hand, research centers produce technical reports and other media 
products that convey important research information and we should have methods for 
knowing to what extent those products are used, how they are used, and if consumers 
view them as of high quality. Therefore, we might well apply research on evaluations 
of service and product quality focusing on match or mismatch between consumer 
expectations and service or product performance (Rust & Oliver, 1994; Oliver, 1977, 
1980, 1989; Parasuraman, Zeithaml, & Berry, 1985; Zeithaml, Berry, & Parasuraman, 
1988; Berry & Parasuraman, 1991). Marketing research shows that when gaps occur 
between customer expectations and service performance, evaluations of customer 
satisfaction are low while if perceived performance exceeds expectations, the customer 
is satisfied (Spreng, MacKenzie, & Olshavsky, 1996). Customers have two levels of 
expectations, adequate service and desired service separated by a zone of tolerance and 
they are oftentimes willing to pay a premium price for high quality service (Bartram & 
Bartram, 1993). Marketing research has also shown a high correlation between 
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executives ratings of whether or not their organizations have reached customer-oriented 
performance outcomes and outstanding financial performance (Whitely, 1991). Other 
marketing studies show that true customers, customers who establish long-term 
relationships with service companies, produce the highest profit margins for the 
company (Berry & Parasuraman, 1991; Zeithaml, et al., 1996). Educational research 
customers may not produce much profit for a research center, but they fit the pattern as 
true customers, that is, they tend to form long term and deep associations and may 
provide an important base of support for grant proposals or when center funding comes 
under attack. 

True customers and the bottom line. The importance of developing deep 
customer relationships is underscored by Louis Gersmer, IBM Chairman and Chief 
Executive Officer. The secret to IBM’s recent soaring stock price according to Gersmer 
and others, is that IBM went back to their original success methods, talking to 
customers, learning their needs, and figuring out how to satisfy them. In the past quarter 
alone, IBM won four of five business contracts it went after and brought in over $1 1 
billion in new computer business (Sager, 1996). James Goodnight’s SAS Instimte, 
weU known to the educational measurement community, is the largest privately owned 
software company with 4,000 employees, 600 million in armual revenues, and a 95% 
lease renewal rate (Lane, 1996). Goodnight, who own two-thirds of SAS, attributes the 
company’s astounding success to its customer service, where Goodnight requires 
employees to write down any suggestion or con^laint and SAS on-line group 
discussions are regularly monitored to detect any gripes or kudos from SAS users. 
Goodnight says of customers service: “It’s an amazingly effective business practice, 
listening to your customers.’’ 

Customer satisfaction standards. While standards are not uniform for 
educational research quality or customer satisfaction, there are numerous other ways in 




29 



which standards might be adapted from customer satisfaction theory and methods. 
Standards may be based on history, i.e., what was last year’s customer satisfaction 
level; engineered, i.e., what should be acceptable satisfaction or performance; market- 
based, i.e., how the competition is doing; planned, i.e., degree to which specific 
objectives are met; subjective, i.e., based on experience, customer comments, and 
managemen t ins tincts; or any combination of the above methods (Juran, 1995). Many 
organizations set a goal to have the two highest performance categories or customer 
satisfaction levels from a quality indicator system reach a theoretical 70% “top box” 
mark (Rodeghier, 1996). Results may be displayed on a Pareto chart, named after 
Vilfredo Pareto, an imp ortant nineteenth century economist credited as the founder of the 
new welfare economic theory (Tarascio, 1968). 

Methods to measure customer satisfaction. For specific tools to 
measure customer satisfaction, education researchers might borrow from the business 
community’s use of a customer satisfaction index (CSI) to provide a best estimate on 
how well an organization is satisfying a specific group of customers (Whitely, 1991). A 
CSI can be composed of multiple measures with 50-60 percent hard measures of 
customer buying behavior and 40-50 percent customer opinions and surveys (Brown, 
1996b). Individual indicators are typically weighted based on organizational goals and 
expectations. S imilar ly, the recently introduced American Customer Satisfaction Index 
(Fomell, et al., 1996) may provide education researchers some ideas for measuring 
customer satisfaction and their own performance as they soon grapple with the 1993 
Government Performance and Results Act. Not only does the ACSI index present a 
theoretical model of customer satisfaction, but specific methodology that might be 
adapted and applied by the education research community to measure its own 
performance. 
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Using databases for profit. Finally, marketing research may have 
crossover lessons for educational researchers in the field of high technology marketing. 
Databases can now be analyzed with sophisticated software to match products to 
specific, even individual prior consumer purchases (Brynes, 1996; Lewington. et al., 
1996). Direct mail advertising can then be quickly designed and sent to individual 
consumers. Education research centers could code their product sales databases and 
notify potential consumers efficiently, even via automated e-mail. 

The foregoing represent only a few ideas that the education research community 
might adapt fiom marketing and consumer research to measure consumer satisfaction 
and the impact of their research. Others are limited only by vision and the desire to get 
educational research to the right people at the time when they most need it (Rothman, 
1980). 

Context f or Present Evaluation Need 

The historical dissatisfaction with the dissemination of education research led in 
1990 to unprecedented pressure on the OERI funded research centers to increase their 
dissemination programs. CRESST, the center that is the focus of this evaluatitm is a 
case in point Although the original 1990 proposal outlined significantly increased 
dissemination activities, the proposal was reuimed to the center containing seven 
negotiation questions. Of those seven, four were related to improved dissemination 
programs, one was related to the evaluation of the program, and one was related to how 
the budget would change to meet the changes in priorities. Therefore only one question 
was completely research related. In response, the Center responded vigorously and 
created an entire program specific to improved dissemination and interagency 
coordination activities. The Center proposed a greater role for its National Advisory 
Board in the evaluation of products which would include practitioner evaluations and 
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feedback cards on all products. A series of impact studies would trace the influence of 
specific research products on practitioners and others. 

This strong response from the Center is indicative of the continued pressure to 
improve dissemination and the benefit that might be derived from better evaluation. The 
importance of dissemination and evaluation is continued in the 1995 CRESST proposal 
for an OERI funded R&D assessment center. Once again, an entire program (one of 
four) is dedicated to dissemination in accordance with the following model (Figure 1). 



Figure 1 

CRESST Dissemination Model 




• Critical Listeiung to the needs and expectations of American communities 
inHiiHing poUcy makers, parents, researchers and teachers to inform R&D and 
the CRESST dissemination program. 



• Research and Development that unfolds to the multiple requirements of 
American communities including policy makers, researchers, practitioners, and 
parents. 




32 



48 



• Interactive Dissemination: Technical assistance, training, or interpersonal 
activities between potential R&D users and providers that oftentimes influences 
beliefs, decision making, and practice. 

• Information Tools: the development or formulation of R&D information into 
a specific tool, product, or presentation that is likely to cause specific action on 
the part of the user. 

• Mass Media: Usually the one-way diffusion or distribution of important 
R&D information via a specific product, i.e. a newsletter, journal article, or 
book. Audiences are large and the intended outcomes are to increase knowledge, 
promote awareness of issues, affect beliefs or attitudes, or encourage “next- 
steps.” 

• Evaluation & Utilization: The evaluation purpose applied to dissemination 
is to collect data that leads to cost-effective aUocation of resources and program 
improvement. For utilization we measure the impact of the overall dissemination 
program or from specific dissemination methods. 

• Reciprocal Feedback is the communication link between each element of the 
model. Thus, Evaluation & Utilization inform aU other parts of the system. 
Likewise, Critical Listening is an ongoing process, continuaUy informing R&D 
and various dissemination strategies. 

The forthcoming evaluation questions derive from the preceding model. 



Evaluation Questions 

Dissemination cannot be “all things to all people (Weiss, 1989).” Nor can 
dissemination mean everything that happens after research is done (HoUifield, 1991). 
Evaluation too has its limitati ons in terms of scope, purpose, and resources (Worthen & 
Sanders, 1987). Recognizing that significant portions of CRESST dissemination 
strategies continue to fall into the “dissemination” paradigm category, the need to limit 
the scope of an evaluation, and to supplement, rather than overlap the 1996 “Impact of 
CRESST R&D” evaluation, this study focused on the evaluation of mass media 
strategies at a national education R&D center although there is crossover and coverage of 
interactive and information tool strategies. This study sought answers to the following 
evaluation questions: 

1 . What do we know about the existing use of CRESST research and its 
diss eminati on products? For example, what products are used, by whom, and 
how often? What, if any, patterns seem to emerge? Who are the primary users? 

2 . How is CRESST research used? What do customers do with the research? To 
what extent does it spread to other users? 

3 . What is the perceived quality and usefulness of CRESST research and products? 
How satisfied are CRESST customers? 

4 . Are there important differences across audiences between perceived quality and 
usefulness of CRESST research and dissemination products? 

5 . What else have we learned about the quality and usefulness of CRESST research 
and products? 
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Why these are hnportant questions. First, while CRESST has anecdotal 
evidence about overall dissemination program effect, little information has been analyzed 
in a systematic way that would lead to valid inferences about use or effects from 
CRESST products. Little is known about product users and use levels, whether or not 
users tend to return for more information (and why or why not) and what might be done 
to encourage increased use. Similarly, knowledge utilization research (Hood, 1972, 
1989) points to the effectiveness of informal, interpersonal chaimels of communication 
as being just as effective as more formal means of communication. Thus, product effect 
may potentially be two or three times the original anticipated effect if products or 
information from those products is spread from one user to another. Program plaimers 
and funding agencies have interest in knowing the extent of such secondary and tertiary 
use. 

Second, acquiring the information aligns with plaimed evaluation activities as 
discussed in the 1995 CRESST proposal. In fact, these evaluation questions are at least 
partially integrated into the 1995 proposal for years 1996-2001. Thus, answers to the 
above questions help to meet program mandates for evaluation. 

Third, answering these proposed research questions may suggest the type of 
products most useful to specific users and lead to important dissemination program 
decisions. The Educational Information Market Study (Hood, et al., 1972) provided 
strong evidence that education research users could be categorized by subgroups and 20 
years later, it remains the most comprehensive review of information use across the field 
of education. Thus, evaluation question 4 attempted to answer whether or not important 
subgroups exist for CRESST research and dissemination products. 

Finally, answers to the proposed research questions may help to inform future 
research needs. Data collection may suggest that self-assessment or assessment for 
special needs students is a vital topic, yet little research has been conducted in this area 
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under earlier CRESST grants. This information may be helpful in subsequent research 
and development years. 



CHAPTERS 



METHODOLOGY 

General Discussion. 

As earlier discussed, very few research designs or instruments have attempted to 
measure the useiiilness of the types of products produced by most research centers, 
including technical reports, newsletters, video programs, conference presentations by 
research center staff, electronic dissemination, etc. The Internet, for example, has 
exploded so quickly onto the scene that most research has been descriptive and 
unsystematic (Berthon, Pitt, & Watson, 1996a). Nor has Internet research produced 
evaluations that might provide possible instruments (Berthon, Pitt, & Watson, 1996a; 
Berthon, Pitt, & Watson, 1996b; Lawrence Rudner, personal communication, 
September, 1995). Fortunately, even though specific instruments are generally 
unavailable, the evaluation questions in this study are by no means beyond common 
evaluation methodology, both quantitative and qualitative. Marketing methods have 
suggested that customer satisfaction surveys can be extremely simple and take up as little 
customer time as possible (Brown, 1996b). 

Product Use Data 

Evaluation Question 1, what do we know about the existing use of CRESST 
dissemination products, was analyzed through a review and synthesis of existing 
CRESST product records including sales records and a center mailing list of over 
14,000 users. Review of existing records frequently provides credible data if existing 
records are reasonably accurate (King, Lyon Morris, & Fitz-Gibbon, 1987). Records 
of product sales at CRESST had been carefully maintained since January, 1992 and thus 
provided a reliable source of data regarding product use and classification of primaiy 
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users. A small pilot study from this data was conducted during the summer, 1996 and 
results reported at the annual California Educational Research Association meeting 
(Dietel, 1996). To supplement existing product orders and better identify audiences, a 
random sample of the entire CRESST mailing list was taken (n=543) using Filemaker 
Pro software. 

Records were also available for CRESST Internet usage and used to provide 
additional insight into program impact and effect. Through their World Wide Web, 
CRESST had developed an interactive registration service (N>800) that provided 
hasftlinft usage data for multiple categories of users including K-12 teachers, principals, 
researchers, parents, principals, etc. Other registration information included how users 
discovered the CRESST Web, i.e. chance, CRESST dissemination, colleague, etc. 
Special software was used to code and produce basic statistical usage rates from the 
CRESST web site between the summer of 1995 to the end of 1996. Finally, a new 
CRESST Web Questiormaire (Appendix C), based on the general design of the 
CRESST Descriptive Questiormaire was used to help answer evaluation questions 2-5. 

Anecdotal evidence suggested that CRESST product sales decrease significantly 
in the summer, however, data had not been previously analyzed to confum this belief 
nor was specific information related to causes for changes in product orders. Records 
were reviewed across specific periods to detect such seasonal trends. Another question 
related to the life cycle of individual products. Once introduced, products oftentimes 
have an expected high number of sales that trail off after a short period of time. But 
some products appear to have much longer life cycles than others without clear 
explanations. Analysis of sales records helped to answer this question. 
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CRESST Descriptive Questionnaire 



Descriptive surveys are useful for gathering a wide variety of evaluation 
information (Alreck & Settle, 1995), especially when the required data cannot be obtained 
as a routine part of program activities or when the size of the taiget group is large and it is 
more economical and efficient to undertake a sample survey than to obtain data on all of 
the participants (Rossi & Freeman, 1989). A descriptive questionnaire was used as the 
primary instrument for evaluating product use, impact, quality, and effectiveness 
(evaluation questions 2-5). A new fastback questionnaire (Appendix A: CRESST 
Descriptive Questioimaire) was adapted from a similar instrument used by the National 
Center for Education Statistics. Use of Educational Research and Development Resources 
by Public Schools Districts (Stalford & Stems, 1990) was a study sent to a specified 
sample of public school districts across the United States concerning their receipt and use 
of research and development resources produced by research laboratories and centers. 
While the NCES survey was limited to only one type of audience, school districts, it was 
a useful starting point for this evaluation of CRESST quality and usefulness. 

As adapted to the purposes of this evaluation, the (ZRESST Descriptive 
Questiormaire covered both major products produced at CRESST, i.e., newsletters, 
media products, technical reports; or by CRESST researchers who authored books or 
made presentations. Because we were interested in discovering signi ficant differences 
between specific users of CRESST research, a number of demographic questions were 
included in addition to several variables thought to infl uence the ratings. The 
independent variables were: 

1 . gender 

2. race 

3 . size of school district (for school districts and schools only) 

4 . elementary, middle, or secondary school (for schools only) 

5 . urban, rural, and suburban schools (for school districts and schools only) 



6 . degree of personal contact with any CRESST staff 

7. receipt of CRESST newsletters 

8. ordering of CRESST products 

9. use of the CRESST World Wide Web 

10. downloading of documents from the CRESST web 

1 1 . sharing of CRESST research or products with others 

The dependent variables were two sets of nine items each (Table 1). The first set 
was intended to measure the overall construct of quality and the second set was intended 
to measure the overall ccmstruct of usefulness of CRESST research and products. In 
order to obtain a broad rating of quality and usefulness, a 1-8 rating scale was used, 
with 1-2 representing poor quality or seldom useful, to 7-8 representing excellent quality 
or very useful (Fink, 1995). 

Table 1'* 

Dependent Variables 

Qu ality Construct-9 Items Usefulness Construct-9 Items 



7. quality of CRESST newsletters 

8. quality of CRESST media products 

9. quality of CRESST technical reports 

10. quality of journal articles or books 

1 1 . quality of presentations 

1 2. quality of CRESST Internet services 

13. overall quality of CRESST research 

14. overall quality of CRESST products 

1 5 . overall coverage of important topics 



16. usefulness of presentations 

17. usefulness of technical reports 

1 8. usefulness of journal articles or books 

19. usefulness of newsletters 

20. usefulness of Internet services 

21 . usefulness of media products 

22. usefulness of ideas by CRESST 

23. overall extent CRESST provided 
you useful information 

24. overall extent useful to the 
education community. 



'* are numbered in the same order as on the questionnaire. The order in which the items were 
presented on the questionnaire was rearranged between each construct to reduce item recognition. 
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Because one of the purposes of the evaluation was to determine if CRESST 
research and products were perceived to have different ratings of quality and usefulness 
from different audiences, the questiormaire was mailed to three specific subgroups, 
users who had ordered products (Product List), Internet registrants (Web List), and the 
full CRESST rolodex (Mailing List). Random samples from each of these subgroups 
(Table 2) were taken from the three appropriate databases. Filemaker Pro software was 
used to generate random lists of names for each subgroup and care was taken to avoid 
duplicate questionnaires to the same person. Further, each subgroup was categorized 
accordin g to three principle audiences who were thought to have different needs and 
who might therefore have different ratings of CRESST research and products. Those 
subgroups were: 1) states and school districts, 2) schools, 3) researchers, and a diverse 
category, 4) others. Because a fair number of parents had registered on the CRESST 
web, they comprised a fifth group for the web only, S) parents. 

Group 1, state and school district users, were primarily assessment, evaluation, 
or curriculum specialists from school districts and state departments of education. Group 
2, the schools category, was generally comprised of K-12 teachers, principals, assistant 
principals, and school counselors. It should be noted that a considerable number, 
perhaps as high as 50% from the schools group appeared to also be graduate students 
using CRESST research for master’s theses or doctoral work. Group 3, researchers, 
included college or university research professors or instructors who taught teachers. 
However, researchers from agencies such as Westat, Educational Testing Service, or the 
regional educational laboratories were included in this subgroup when appropriate. 
Group 4, others, was comprised of many non-profit or for-profit agencies involved in 
educational work or firom agencies such as AFT, NEA, National Forum on Assessment, 
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etc., with a strong interest in educational reform and improvement. Media, test 
publishers, and corporate audiences were also in group 4. 



Table 2 

Registration Type and Audience Groups 



Registration Group 


Product List 


Web List 


Mailing List 


Group 1 


States/Districts 


States/Districts 


States/Districts 


Group 2 


Schools 


Schools 


Schools 


Group 3 


Researchers 


Researchers 


Researchers 


Group 4 


Others 


Others 


Others 


Group 5 


Parents 







PUot and full study. The CRESST Descriptive Questionnaire was pilot 
tested in late Spring and Summer, 1996 prior to full scale distribution in the Fall, 1996, 
with no significant problems encountered. The final Descriptive Questionnaire was 
identical to the pilot questionnaire but mailed to new stratified random samples from the 
same lists; therefore, results from both the pilot and final study were combined. In the 
pilot study, 392 questionnaires were sent out and 216 returned for a response rate of 
53%. Two follow-up letters were sent out in order to produce the final return rate. For 
the final study, 2051 questionnaires were mailed out with one follow-up letter and 875 
questioimaires returned producing a total response rate of 43%. Across the three 
different registration types, response from the product list was 290 of 655 (44% 
returned), fiom the web list, 277 of 608 (44% returned), and from the center rolodex 



mailing list 308 of 788 (39% returned). Approximately 5% of the questionnaires were 
returned unopened with bad addresses or individuals who had retired or moved. 

Robust sample size. T-tests of the means for each item between the pilot 
study and the fiiU study did not reveal any significant differences between responses, 
suggesting that the somewhat lower response rate for the fiiU study did not create 
serious bias and did not require the expense or time of a third foUow-up mailing for the 
final study. It also supported combining the data from the pilot and full study to: 1) 
produce robust data analysis and accurate inferences; 2) to account for missing 
responses to individual items on the questionnaire; and 3) to anticipate complete deletion 
of those questionnaires with so few items completed as to create questions as to the 
dependability of the questionnaire. Approximately 150 questionnaires were dropped as a 
result of incomplete data returned. The final sample size for analysis was 751; however, 
this is the total number of valid questionnaires, total responses for individual items are 
generally lower. 

In order to achieve a maximum response rate, the CRESST Descriptive 
(^estionnaire was limited to two pages, plus a cover letter, similar to the NCES survey 
earlier discussed. The total number of questionnaires mailed was based on the results of 
the pilot test, slightly oversampling the groups and subgroups with lower response rates 
from the pilot study. 

CRESST Product Questionnaire. 

Telephone interviewing can provide data relatively free from bias and offers 
advantages of economy and time versus in-person interview techniques (Babbie, 1992). 
Therefore, to help answer other research questions and provide more in-depth data, a 
telephone protocol was developed and pilot tested in late Spring, 1996. The instrument 
contained items on how CRESST products and research were used (Evaluation (^estion 
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2), items related to customer satisfaction levels (Evaluation Question 3), and several 
general questions about CRESST research and products (Evaluation Question 5). 
Throughout the pilot, a problem surfaced in reaching customers. Also, information too 
closely overlapped the CRESST Descriptive Questionnaire and was producing similar 
responses; consequently, questions were modified to answer additional questions about 
the use of CRESST products, including how users found out about products, purpose 
of order, timftlinftss in product arrival, best way to receive notification and receive 
products, likeliness to order products again and recommend to others, and time of year 
that was best to order products. Each of these questions could provide information 
useful to the future marketing of CRESST research and products. 

A revised telephone interview protocol was used for the full study and 16 
telephone interviews conducted. However, because it continued to be time-consuming, 
and therefore, expensive to contact at least 30 participants by telephone, the interview 
protocol was converted into a descriptive product questiormaire and mailed to a random 
sample of product users (n=147). No follow-up mailing reminders were sent. Fifty-two 
product questiormaires were returned for a 35% return rate and coded into SPSS. For 
identical items between instruments, responses were combined into a single sample, 
creating a final response sample size of 68. The questiormaire was again lirmted to a 
sin g le page, front and back, in order to ensure a high return rate. 

CRESST Web Questionnaire 

A telephone questiormaire was also conducted of CRESST Internet users and 
went through a similar transformation process as the Product Questionnaire. We were 
most interested in how Internet users found out about the CRESST World Wide Web, 
the types of information that they utilized from the site, how frequently they visited the 
site, how many major products they downloaded, i.e., technical reports and newsletters. 
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what type of information was sought on the C31ESST web site and to what degree they 
found what they needed, extent and type of use of the information, what additional 
information was needed, and likeliness to use the site and recommend it to others. These 
questions were related to all five evaluation questions. Again response to these 
questions via telephone format produced a smaller number of responses than desired 
(n=16) and a CRESST Web Questionnaire was developed (Appendix Q and mailed to a 
new random sample of web registrants (N=158). Thirty-eight questionnaires were 
returned for a return rate of 24% and again, where items from both questionnaires were 
identical, responses were combined resulting in a total sample size of 54. Parents did not 
respond in adequate numbers and were coded as “other." Using the CRESST web site 
as a possible source for questionnaire response was considered but not implemented 
because the sample would not meet requirements for randomness and because of 
concern for detrimental impact on the existing web survey that provides basic 
demographic data on CRESST web registrants. 

Data from the CRESST World Wide Web server was logged across specific time 
periods focusing on usage rates and also the download of CRESST portable document 
format (PDF) reports and newsletters. 

Short Response Items 

Although re-designed to provide mostly quantitative data, both telephone 
questioimaires, CRESST Product/Resources and CRESST Web Site, contained short 
answer, qualitative data that was transcribed, coded, and analyzed in accordance with 
appropriate qualitative analysis methods (Erickson, 1986; Miles & Huberman, 1984; 
Smith, et al., 1994; Strauss, 1993). The unit of analysis was defined as a product user 
for the CRESST Product Questionnaire and a CRESST Internet Web registrant for the 
CRESST Web (^estionnaire. Because nearly all of the qualitative entries were less than 
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255 characters, SPSS 6.1 was used for data entry with responses simultaneously coded 
quantitatively. The same procedure was followed for qualitative sections of the 
CRESST Descriptive Questiormaire and CRESST Web Site Questiormaire. 

Validity of the CRESST Descriptive Questionnaire 

One of the purposes of this evaluation was to begin to develop evaluation 
instruments that could be used across the current five years of CRESST research and 
beyond. The instruments could also be useful to other education research centers and 
laboratories. In order to establish a measure of instrument reliability, statistical tests 
were performed to measure internal consistency of items measuring each construct, that 
is, quality and usefulness. A factor analysis was run to determine if the items appeared 
to be measuring the intended constructs. 
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Chapter 4 
FINDINGS 



Evaluation Question 1. What do we know about the existing use of 
CRESST research and dissemination products? 



General Discussion^ 

To narrow down this very broad question, we subdivided it into several other 
questions as follows: who typically uses CRESST research and products and where are 
consumers distributed geographically; how do consumers “find” CRESST research; 
what research do consumers use and approximately how much; and finally, what does 
the life cycle of a CRESST product look like? 

Who uses CRESST research and dissemination products? 

To answer this question an analysis of product records was completed from 
three sources: the full CRESST mailing list of over 14,000 CRESST customers, 
CRESST product orders from 1992-1996, and CRESST web registration digests. 
Additionally, records from the CRESST web site were collected and analyzed to 
determine usage rates, types of products used, and types of users. 

CRESST Consumers by Gender. A random sample of 543 names from the 
CRESST mailing list of 14,323 names produced the gender results found in Table 3. 
Gender showed a slightly higher female representation, 53.5% to 46.6%. Missing 
gender data from the mailing list resulted because 52 of the 543 cases had only last 
names on the CRESST mailing list, a last name with only an initial for a first name, or 
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first nampR that could not be identified on a gender basis, i.e., Pat Smith, Sang Kim, 



etc. 



Tables 

Gender of CRESST Users from Center Mailing List 



Source 


Gender 


Frequency 


Percent 


Cum Percent 


Mailing List 


Female 


262 


53.4 


53.4 


n=491 


Male 


229 


46.6 


100.0 


Web List 


Female 


66 


54J. 


54.1 


n=122 


Male 


56 


45.9 


100.0 


Product List 


Female 


90 


64.7 


64.7 


n=139 


Male 


49 


35.3 


100.0 



Similar demographic information was collected fipom a random sample of the 
CRESST web list (n=122). For gender, females (54. 1%) were slightly higher than 
males (45.9%). Product user information was derived from 1992-1996 product orders. 
Females (64.7%) ordered more CRESST products than males (35.3%), however it must 
be noted that over 50% of CRESST products are bought through a purchase order and 
man y of these are ordered by administrative staff who may be female to an unbalanced 
degree and thus not a good indicator of the gender for the real consumer. 

CRESST Research Consumers by Occupation. Occupations for 
CRESST users were coded according to the categorization scheme in Table 4. It is 
possible that individuals could belong to two or more audience groups. While care was 
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taken to code eveiy person consistently, some bias was likely to occur and results 
should be interpreted accordingly. 

Consumers from the local government category, 16.4%, 11.7%, and 31.7% 
across registration lists, were primarily test, research, and evaluation specialists at the 
school district level, members of local schools boards or county boards of education. 
The University category, 14.5% for the Mail List, 23.4% for the Web List, and 22.8% 
for the Product List, usually included education research professors or professors who 
taught teachers. State goverrunent employees, 13.7%, 6.3%, and 2.7% respectively 
across the three lists were usuaUy members of goverrunent departments of education or 
state education legislative analysts. Researchers, 7.7% for the product list, were 
generally R&D specialists from the national educational laboratories or from 
organizations such as the American Institutes for Research that specialized in education 
research. The K-12 Other group, 7.4% for the CRESST mailing list, were principals, 
school counselors, or other K-12 school specialists. The non-profit education category 
included organizations such as the American Association of School Administrators, 
NEA, AFT, etc. Unknown organizations, 17.7% for the CRESST mailing list and 
1 1.7% for the product list, primarily occurred because names were listed without any 
organizational affiliation. 

For organizational affiliations from the CRESST web site, universities were the 
highest of any group (23.4%), perhaps reflecting their excellent Internet access, 
followed by K-12 teachers (18.7%) and local government employees (1 1.7%). If both 
K-12 teachers and K-12 Other groups were combined, K-12 employees would surpass 
any other group (24.2%). Local goverrunent employees were by far the most frequent 
CRESST product users (31.7%) followed by university employees (22.8%). No other 
users exceeded 10%. The high rate of purchase from school districts is substantiated by 
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physical review of purchase orders and other anecdotal evidence provided by the 
administrative assistant who handles CRESST product sales. 

Table 4 

Organizational Affiliation of CRESST Research and Product Consumers 



Occupation 


Mail 

list 

Frq 


Mail 

List 

Percent 


Web 

List 

Frq 


Web 

List 

Percent 


Prod 

List 

Frq 


Prod 

List 

Percent 


Local Gov. 


89 


IM 


15 


1L2 


46 


31.7 


University 


79 


14*5 


30 


23.4 


33 


72A 


State Gov. 


74 


13.7 


8 


6.3 


4 


2.7 


Nonprofit ed 


49 


9.1 


4 


3.1 


0 


0 


K-12 Other 


40 


lA 


7 


5.5 


6 


4.1 


Media 


24 


4.4 


0 


0 


0 


0 


Corporate 


23 


4.2 


11 


8.5 


5 


3.4 


Fed Gov. 


18 


3.3 


0 


0 


0 


0 


Research 


17 


3.1 


7 


5.5 


11 


LI 


Test Pub 


10 


1.8 


0 


0 


0 


0 


Library 


10 


1.8 


5 


4.0 


5 


3.5 


K-12 teacher 


7 


1.3 


24 


mi 


6 


4.1 


Other 


7 


1.3 


8 


6.3 


12 


8.3 


Parent 


0 


0 


9 


7.0 


0 


0 


Unknown 


96 


nj 


0 


0 


17 


ILI 


Total 


543 


100 


128 


100 


145 


100 
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Geographic locations of consumers. Geographic categories of CRESST 
research users from the center mailing list (Table 5) were based on the basic distribution 
of states within the regions of the 10 national educational laboratories. Our finding was 
that the largest concentrations of individuals on the CRESST mailing list reflect similar 
populations as those of the regional laboratories with large concentrations in the Far- 
West including California (24.3%), mid- Atlantic states (18.4%) including New York, 
and the North Central (15.1%) area, including Blinois. 

Tables 

Geographic Location of CRESST Users from Center Mailing List 



Region 


Frequency 


Percent 


Far West 


132 


24.3 


Mid-Atlantic 


100 


18.4 


North Central 


S2 


15.1 


Southeast 


62 


11.4 


Northeast 


46 


8.5 


Mid-Continent 


46 


8.5 


South-central 


30 


5.5 


Northwest 


27 


5.0 


Appalachia 


9 


1.7 


Pacific Islands 


7 


1.3 


Unknown 


2 


.4 


Total 


543 


100.0 
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Web and Product Registration Lists. The CRESST Web site registrant 
list showed the highest use rate from Mid-Atlantic (18.4%) and North Central (15.2%) 
regions followed by the Far West (13.6%), Foreign (12.8%) and Mid-Continent regions 
(8.0%). Missing geographic locations (2.3%) were a result of inadequate state or 
country origins that CRESST web registrants failed to complete on the CRESST web 
registration list. Geographic location for product users followed similar trends as the 
web registration list and the CRESST mailing list with the Mid-Atlantic region 
accounting for 14.5% of product usage, North Central, 20.7%, Far West, 16.6%, and 
Mid Continent, 13.8%. Our conclusion is that CRESST research usage is fairly evenly 
distributed across the United States. 

How do customers discover CRESST research and^roducis? 

Item 6 from both the CRESST Product Questionnaire and the CRESST Web 
Questioimaire queried consumers on how they discovered CRESST products or the 
CRESST web site. While this is not an indicator of how all CRESST research is 
discovered, it does provide an indicator for a substantial number of customers (Figures 
2 and 3). 

Product Order Discoveries. CRESST product registrants most often found 
products through CRESST publications (40.3%). Printed mention in non-CRESST 
publications accounted for 13.4% of discoveries, colleague or professor, 11.8%, and 
more than one source, 11.8%. 

Web Site Discoveries. CRESST web registrants found CRESST most often 
through a general Internet web search on education topics (28.8%) and general Internet 
web search on testing topics (21.2%). Middle level categories for finding the CRESST 
web site were from other web site providers (13.5%) and from printed mention in a 
CRESST publication (11.5%). Several respondents on the CRESST web questionnaire 
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said they found out about CRESST research and products from presentations by one of 
the CRESST co-directors or the CRESST associate director. 

Figure 2 

How CRESST Product Customers Discovered CRESST Products 



n-68 




How found products 



/ 
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Figure 3 

How Web Registrants Discovered the CRESST Web Site 



n-54 



40i 




How many research products do C RESST customers wse2 

Products Sales. To help answer this question, overall CRESST product sales 
were tracked from 1992-1996 and are reported in Table 6 and Figure 4. CRESST 
product sales underwent a period of growth in 1992 and 1993, peaking in 1994 and 
decreasing slightly in 1995, and in 1996, sales dropped to 1992 levels. In 1995, 
CRESST made approximately 70 technical reports of its most recent research available 
from their Internet site in a portable document format (PDF) and at no cost to users. A 
PDF document prints out identical to a fully formatted CRESST technical report 
including charts, tables, and figures. Additionally, CRESST produced a CD ROM in 
early 1996 that contained over 75 CRESST technical reports, a number of assessment 
guidebooks, test items, and two videos. These two dissemination efforts have clearly 
contributed to the observed reductions in CRESST product sales, as evidenced by e-mail 
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messages from CRESST web users requesting CRESST products over the Internet vs, 
purchasing them. However, total CRESST dissemination has increased (see Internet 
usage to follow). Average CRESST product sales are based on a 30-day month. The 
daily average product sale across all five years is S.2 products/day based on a 365 day 
year. 
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Table 6 



CRESST Product Sales, 1992-1996 



Month 


1992 


1993 


1994 


1995 


1996 


Mean 


Jan. 


45 


270 


120 


306 


113 


171 


Feb. 


59 


212 


291 


254 


117 


187 


Mar. 


176 


56 


155 


248 


73 


142 


Apr. 


225 


242 


451 


341 


177 


287 


May 


107 


210 


232 


169 


145 


173 


Jun. 


141 


132 


286 


180 


219 


192 


Jul. 


110 


166 


184 


121 


145 


145 


Aug. 


58 


164 


105 


68 


110 


101 


Sep. 


72 


141 


100 


31 


186 


106 


Oct. 


127 


64 


151 


258 


126 


145 


Nov. 


171 


92 


149 


163 


79 


129 


Dec. 


332 


80 


28 


83 


70 


121 


Mean 


135 


m 


188 


185 


130 


158 


Total 


1623 


1829 


2252 


2222 


1550 
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Figure 4 

CRESST Product Sales 1992-96 
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Note from Table 6 and Figure 4 the high variance in products purchased between 
months with a persistent peak for product purchases in April of each year and slower 
sales in August, September, and December. These trends were further investigated in 
the CRESST Product Questiormaire (See findings from Evaluation Question 5). 

CRESST Web Usage. CRESST Web usage was tracked from the late 
summer, 1995 to the end of December, 1996. Results are reported (Table 7) in terms of 
much more conservative web site “visits” instead of the mote commonly used “hits.” A 
hit is any cormection to an Internet site, including inline image requests and errors, thus, 
an inaccurate estimate of true web usage. Visits on the other hand, are a series of 
consecutive requests from a user to an Internet site. Tracking CRESST hits vs. visits 
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indic ates that, in general, visits are approximately 1/6 the number of hits, thus 100 daily 
visits could be approximately 600 daily hits. Only visits are reported here and the 
results from internal visits, that is, usage by internal CRESST web managers, are 
excluded. There are some overlaps between periods based on when logs were analyzed. 
The accuracy of any web usage is at best an approximate estimate; however, the data 
collected suggest a relatively stable usage rate with fairly small changes over all periods 
reported. 



Table? 

Average Daily Visits to CRESST Web 



Period 


Dates 


Average Daily Visits 


1 


8/10/95-12/18/95 


115 


2 


12/18/1995-2/26/96 


99 


3 


2/26/96-5/5/96 


125 


4 


5/5/96-6/14/96 


97 


5*’ 


6/7/96-12/11/96 


122 


6 


10/22/96-12/31/96 


109 


Meat?" 




ill 



Within the last year, CRESST switched to new statistical software that includes a 
summary of average duration of a visit. An analysis of three periods for which data were 
available suggests that visitors may be connecting for longer periods of time. In the fall 



'* Note that there was overlap between periods 5 and 6, a result of the tracking system on the CRESST 
web site. 

“ Although a mean is computed for the average of CRESST web site usage, the periods are unequal and 
thus the mean is, at best, a rough approximation. 
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of 1995, visitors averaged 2.29 minutes connected to the CRESST web while fiom 
June, 1996-December, 1996, the average time connected was 6:01 minutes. 

PDF Documents. Since early, 1995, CRESST has provided their research 
reports in a downloadable format with perfectly formatted tables and graphics via the 
portable document format (PDF). Table 8 shows the number of PDF documents 
downloaded from the CRESST web site for Periods 1, 2, 3, 4, and 5. Approximately 
75% of the CRESST web PDF files are CRESST technical reports while other PDF 
documents are newsletters, handbooks and guidebooks, overheads from major 
presentations, or other types of CRESST research. All are major size documents and 
may take, anywhere from a few seconds to download to a few minutes depending on the 
users’ type of cormection. 

Tables 

Average Daily PDF Files Downloaded from CRESST Web 



Period 


Dates 


Average Daily Downloads 


1 


8/10/95-12/18/95 


18.7 


2 


12/18/1995-2/26/96 


16.2 


3 


2/26/96-5/5/96 


17.2 


4 


5/5/96-6/14/96 


15.0 


5 


10/22/96-12/31/96 


32.4*' 


Mean^^ 




19.9 



32.4 was a very conservative estimate based on both technical reports downloaded and newsletters. 

^ Although a mean is computed for the average of CRESST PDF documents, the periods are unequal and 
thus the mean is at best an approximation. 
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Because CRESST sells an average of 5.2 products per day in a 365 day year, the 
downloading of CRESST reports and other products more than triples total products 
distributed, based on an average of 20 PDF documents downloaded per day. As 
mentioned before however, CRESST product sales have decreased over the last year, 
approximately 2 fewer reports are sold per day. A sample downloading log of PDF 
documents is found in Table 9. 



Table 9 

S amp le of PDF Files Downloaded 
from CRESST Web (10/22/96-12/31/96) 



Product Title Download 



416 Assessing the Validity of the NAEP: The White Pjqjer 209 

415 Performance Puzzles: Issues in Measuring Capabilities. . . 182 

410 Issues in Portfolio Assessment: Scorability of Narrative. . . 166 

414 Evidence and Inference in Educational Assessment 150 

371 Can Portfolios Assess Student Performance and Influence. . . 134 

402 Monitoring and Improving a Portfolio Assessment System 103 

391 A First Look: Are Cl aims for Alternative Assessment. . . 91 

397 An Analysis of Parent Opinions and Changes in Opinions. . . 86 

362 Performance-Based Assessment and What Teachers Need 86 

348 Accountability and Alternative Assessment 82 
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The preceding table shows not only the popularity of the CRESST web site for 
downloading documents, but indicates that consumers are willing to search in order to 
find the reports they are interested in. Technical reports 362 and 348, for example, are 
from 1993 and 1992 respectively, yet they are still being downloaded at a rate of one per 
day. This table, combined with the product sales logs and other web PDF logs show 
that alternative assessment (also caUed performance-based assessment) and portfolios 
remain very popular topics. 

CRESST P roduct Life Cycle 

In order to review the life cycle of a (TRESST product, quarterly product sales 
were tracked for four different products (Table 10): a handbook, the CRESST 
Performance Assessment Models, Content Area Explanations; one videotape. Portfolio 
Assessment and High Technology; and two (TRESST technical reports, 334, Effects of 
Standardized Testing on Teachers and Learning: Another Look, and report 355, The 
Reliability of Scores from the 1992 Vermont Portfolio Assessment Program. 

The products were relatively diverse in both content and price. The handbook 
features a well-known CRESST model for developing performance assessments and 
was the top selling product for nearly three years, while the videotape is an overview of 
a CRESST portfolio assessment research project at an elementary school. The technical 
report titles are self-explanatory. The most expensive product was the videotape, 

$15.(X) including a guidebook on portfolios, and the least expensive was the Vermont 
portfolios technical report at $3.00. In general, price does not appear to be a major 
detriment to product purchase due to low price and product uniqueness. 
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Table 10 

Average Quarterly Sales of Four CRESST Products 



Prod # 


Product Title 


Avg. Qtr. Sales 


Hand 


CRESST Performance Assessment Models. . . 


37.6 


V2 


Portfolio Assessment and High Technology 


21.6 


TR334 


Effects of Standardized Tests on Teachers and. . . 


11.1 


TR355 


Reliability of Scores fiom the 1992 Vermont. . . 


8.3 



While all four CRESST products showed similar sales trends, that is, a steep 
initial peak in sales usually followed by quick decline, there are some obvious 
differences (Figures 5-8). Both the CRESST Models handbook and the portfolio 
assessment videot!q)e have higher mean sales and a longer active life-span. It seems 
reasonable to su rmis e that the reason for these differences are that the CRESST 
handbook and videotape are relatively unique products compared to the two technical 
reports, and have relevancy or potential use for an extended period of time. Only one 
CRESST assessment models handbook exists and only two CRESST videotapes, while 
dozens of other technical reports are available. In particular, report 355 had to compete 
with as man y as five other reports on the same general topic, the Vermont portfolio 
assessment program, first-authored by the same researcher in each report. A review of 
the product sales records, however, indicated that when one Vermont report was 
purchased, others were oftentimes purchased simultaneously. 
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Figure 5 

Life Cycle: CRESST Perfonnance Assessment Models, Content Area Explanations 



CRESST Product Life Cycle 




QTR 



Figure 6 

Life Cycle: Portfolio Assessment and High Technology Videotape 
CRESST Product life Cycle 
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Figure? 

Life Cycle: CRESST Technical Report 334 
Effects of Standardized Testing on Teachers and Learning: Another Look 
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Figure 8 

Life Cycle: CRESST Technical Report 355 
The Reliability of Scores from the 1992 Vermont Portfolio Assessment Program 
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Evaluation Question 2. How are CRESST research and products 
used? 



General Discussion. 

To answer this question, we analyzed to what extent CRESST customers read 
and reviewed CRESST materials purchased or downloaded from the CRESST web, 
what customers reported that they did with the research, to what extent the research 
appeared to spread to secondary users, and finally, what appeared to be the impact from 
its use. 

Findings from extent read and reviewed. 

Question 8 from the CRESST Product Questiormaire and question 13 from the 
CRESST Web Questionnaire asked the question “to what extent did you use any 
materials viewed or retrieved?" Use was defined as the extent that materials were read 
and reviewed. Because significance differences were not found between the CRESST 
Product Order or Web Questiormaires [t(48)=.20, p=.844], data were combined 
(n=122) for Table 11. 
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Table 11 

Extent CRESST Research Was Read or Reviewed 



Read or 
Reviewed 


Frequency 


Percent 


Valid Percent 


Cum 

Percent 


< 10% 


13 


10.7 


11.3 


11.3 


about 10-50% 


37 


30.3 


32.2 


43.5 


about 50-75% 


32 


26.2 


2L£ 


71.3 


over 75% 


31 


25.4 


27.0 


98.3 


Don’t recall 


2 


1.6 


1.7 


100 


Missing 


7 


5.7 


Missing 




Total 


122 


100.0 


100.0 





Just over 32% of respondents answered that they read or reviewed the research 
between 10-50% with nearly 55% reporting that they read or reviewed over 50% of the 
research material. While comparison figures are not available from other education 
research centers, this appears to be a fairly high level of reading or reviewing of products. 

Findings from how CRESST Products a nd research are use<L 

Purpose of Product Order. Question 7 from the CRESST Product 
Questiormaire asked customers “what was the primary purpose of your [product] 
order?” Respondents (n=68) said that the purpose (Figure 9) was for research (34.9%), 
to inform policy (21.7%), or to inform teachers (12.8%). Test development (1 1.3%) 
and student research (6.6%) were the other primary purposes mentioned. Note that 
respondents were given the opportunity to list two primary purposes. Answers were 
combined into a single variable, purpose of use, and 38 of 68 respondents listed more 
than one purpose for their product order. 
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Figure 9 

Purpose for Ordering CRESST Products 



n-68 

40 T 
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Inform Policy Tot Devdapnicnt Other 
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Use of CRESST research. Use of research from the CRESST Product 
Questio nnaire is reported in Figure 10, based on answers from the question “what did 
you actually do with the information viewed or retrieved.” Although this item was open 
ended, it was coded based on the ..purpose of your [product] order” question. 

Results showed equal percentages of sharing with teachers (24.6%) and for background 
information (24.6%), followed by formulating policy (13.1%). Just over 21% of 
respondents used CRESST research for more than one purpose. 



Figure 10 

What Customers Did with CRESST Research 



n-61 




What customers did with CRESST rescach 



Findings on use from open-ended questions. Results from open-ended 
questions on the CRESST Web and Product Questionnaires provide more specific 
insight into the uses of CRESST research. The following responses are fix>m the open 
ended question “What did you do with the information you reviewed or retrieved?” fix>m 
the CRESST Product Questionnaire: 

• Coordinator at National Center on Educational Outcomes shared CRESST 
materials with Staff and added them to their center database. 

• A senior research scientist at Haicourt, Brace Educational Measurement shared 
CRESST materials as a Sample of how to effectively present materials. 



* Director of Special Projects at CUNY used CRESST research for a doctoral 
dissertation and development of published material on whole language for 
adults. 

* A researcher used CRESST materials for general information, for research, and 
as policy information for policy makers. 

* Director of English learning center used CRESST research to compare ways 
others are assessing language students, especially writing. * 

* A Florida state university professor used CRESST research to teach an 
assessment course and in his own professional writing. * 

* President of Cooperative Ventures used CRESST research for training a charter 
school in alternative assessment strategies. * 

* Director of assessment for a school district used CRESST research during 
facilitation of secondary assessment task force in school district. * 

* A school principal incorporated CRESST research into assessment plaiming for 
his school. * 

* represents second-level use. 



The foUowing are examples from the open-ended question on the CRESST Web 
Questiormaire: What did you do with the information you viewed or retrieved [from the 
CRESST web]? 

• West Ed researcher used CRESST information to provide background for a case 
study involving leveraging the curriculum through testing. 
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• A high school teacher used CRESST (web) materials to share with fellow staff 
members at West Hawaii Explorations Academy/Konawaena High School. 

• A reference librarian at the University of Texas bookmarked CRESST web site 
for use in helping students find curriculum standards. 

• A teacher used CRESST research for state accreditation purposes. * 

• Assistant principal at a high school shared CRESST research with teachers and 
used it in his curriculum development efforts. * 

• Assessment director for school district used CRESST research to shape local 
[assessment] decisions and distributed to others. * 

• A psychologist integrated CRESST research into practice and disseminated to 
other professionals. * 

• The director of evaluation and assessment at the Tennessee Department of 
Education used CRESST research to inform development of the [Tennessee] 
state assessment program. * 

• represents second-level use. 

While many diverse uses were reported on the open-ended questions, a pattern 
seemed to appear. CRESST research was used as a general information tool, i.e., 
shared with others, or used as both an information tool and also taken to a higher level, 
that is, integrated into policy or practice. Based on this pattern, the open-ended items 
ftom both the CRESST Product and Web Questionnaires were coded into information or 
integrated uses. An asterisk (*) at the end of each preceding example illustrates the types 



of cases that were coded as strictly informational use vs. an integrated use. Coding 
from all questionnaire responses resulted in 23 of 66 examples of CRESST research 
(35%) that were integrated into policy or practice. All items were coded as informational 
at minimum. 

To what extent does CRESST research and its products spread to 
secondary users? 

All three instruments in this evaluation contained questions related to sharing of 
CRESST products and research with others. Table 12 represents the percentage of 
customers and the level they reported of sharing CRESST research from aU three 
questionnaires. In order to account for the much higher number of respondents to the 
CRESST Descriptive Questionnaire, a weighted average in the final column averages 
sharing CRESST research across all three questionnaires but with the descriptive 
questionnaire receiving four times the weight of each of the other questionnaires 
separately. 



Table 12 

Percent of CRESST Customers Who Shared CRESST Research: 
Results From Three Questionnaires 



Shared Research 


Web Quest. 


Product Quest. 


Desc. Quest. 


Weighted Avg. 


Never 


21.0 


2.0 


12.0 


12.0 


Once 


18.5 


19.0 


14.0 


15.3 


2-3 times 


39.5 


46.0 


43.0 


43.0 


4 or more times 


18.5 


29.0 


30.0 


2M 


Unsure 


2.5 


4.0 


1.0 


1.7 




100.0 


100.0 


100.0 


100.0 



The weighted averages indicate that only about 12% of CRESST audiences did 
not share CRESST research at least once within the last two years. About 15% shared 
once, 43% shared 2-3 times, and 28% shared 4 times or more. Again, while 
comparative data is not available, this seems to represent a very high degree of sharing. 

Another question related to the sharing of CRESST research was the question on 
both the CRESST Product and Web Questionnaires “How likely are you to recommend 
CRESST to others?” discussed in the findings of Evaluation Question 3. 
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What is theJmpact from the use of CRESST research and its 
dissemination products? 



Data to answer the question of Q^ESST impact came from a combination of 
sources including comments from the Q^ESST Web Registration Digests, telephone 
interviews, e-mail inquiries via the CRESST Web “Ask-Us Service,” review of a 1996 
evaluation report ‘The Impact of CRESST R&D” (House, 1996), open-ended items 
from all three CRESST questioimaires, and various CRESST memos and documents 
including the 1995 CRESST Impact Report to the Office of Educational Research and 
Improvement. Data was coded in accordance with acceptable qualitative procedures. 
The range of CRESST impact was found to be enormously diverse and is perhaps best 
categorized at specific ranges of influence, from simple information at the individual 
level to broad state and national level irrqract. Much of CRESST’ s impact is transparent 
since it is usually filtered by various change agents, i.e., researchers, professors 
instructing new teachers, measurement experts, testing professionals, school district and 
state test directors, and policy makers. The patterns for impact that emerge from an 
overview of CRESST records can be categorized into the following: impact on practice, 
impact on measurement theory and research, impact on policy with implications for 
practice. 

The following exan^les, representing interactive dissemination, information 
tools, or mass media strategies from the CRESST dissemination model, were selected as 
appropriate representations for each major category of CRESST impact. 
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Impact on practice. 

• Long time, no talk, but I didn’t forget you. I took your advice, and over the last 
several months we have focused on changes in teacher behavior that 
performance based learning and assessment can provoke. We did an extremely 
well received workshop with Mike Hibbard from Connecticut and four of his 
teachers. Our staff has gotten so excited that we’ve had two follow-up meetings, 
each with new tasks, assessments lists, and student work, and we’re having a 
thir d in a couple of weeks where they’re each bringing two friends. It’s catching 
on because teachers see the utility, alignment with their own efforts, and 
empowerment for themselves and their kids. 

E-mail message 

School District Assistant Superintendent, California 

• I enjoyed talking with you and getting your view of the [assessment] issues we 
are exploring. Thanks you very much for sending copies of CRESST’s “Final 
Report: Perceived Effects of the Maryland School Performance Assessment 
Program” and for calling me with Joan Herman’s information. Your responses 
will be extremely helpful to us in the work we are doing on assessment. 

Letter from the 

Assistant Director, American Federation of Teachers 

Impact on measurement theory and research. 

• In an examination of 35 [ERIC] articles, CRESST research was cited 90 times in 
CRESST authored publications and 42 times in 26 non-CRESST articles. Most 
CRESST articles were in the highest status journals. All in all, the evidence is 
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extensive that CRESST has had a major impact on the measurement research 
corrununity. Ernest House 

The Impact of CRESST R&D (1996) 

Impact on policy with indirect impact on practice. 

• I’m using it [seven CRESST research products] for a school reform committee, 
I’m chairman of the goals and testing conunittee for the state. I also use the 
iTifltftri ak when I consult with districts and often copy the materials and give 
them to others. We are working on performance assessments statewide in 9-12, 
a series of tasks, and wanted to find out what was going on in Arizona [CSE 
Reports 373, 380, 381]. We wanted to avoid those types of problems. Our 
program will be voluntary, not mandatory. 

Telephone Interview 
Professor, University of Idaho 

• Last spring I contacted you about our district's Certificate of Initial Mastery 
Tests. Our school board recently rescinded its previous policy making these exit 
exams a graduation requirement and I thank you for providing me the 
information necessary to keep us from a potentiaUy serious lawsuit. 

E-mail message fiom English high school teacher in Maine 



^^This response went well beyond the question suggesting significant impact from CRESST research based 
on products. The same respondent said that CRESST is the only place he gets useful [assessment] 
information and that CRESST funding needs to continue. When asked of improvements, he said he would 
just love to have CRESST closer so he could come see them. As a result of the telephone interview, he 
said he wanted to attend the next CRESST conference. 
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It should be noted that each of these cases exemplify the important role of change 
agents in the dissemination process and that most CRESST impact seems to occurs 
through intermediaries, oftentimes reinforcing existing opinions or in some cases, 
providing evidence to support a specific point of view. 

Broad impact on policy at national, state, and local levels with major 
implications for practice. 

We found significant instances of impact on national, state, and local education 
agencies that at times could be attributed to CRESST as an organization, or in other 
cases, more directly linked to individual CRESST researchers. As stated during our 
discussion of the CRESST dissemination model, we focused on CRESST as an 
organization, with a clear understanding of the important leadership role played by 
center directors and other influential researchers that comprise the CRESST paitners. 

National Impact An example of the influence exerted by CRESST leadership 
was the appointment in 1991 of CRESST co-director Eva Baker to the National Council 
on Education Standards and Testing. The Council had been convened to advise 
Congress on the “desirability and feasibility of national standards and tests, and [on] 
recommendations for long-term policies, structures, and mechanisms for setting 
voluntary education standards and planning an appropriate system of tests (National 
Council. . . 1992).” As chair of the Assessment Task Force, Baker successfully 
advocated for man y of the council’s final recommendations, including continued use of 
the NAEP to monitor student progress to national standards and that results from any 
national assessment system only be used for instmctional purposes until validity issues 
were resolved. Of added note was that the Council’s report helped to dampen the 
impetus towards the development of a new national test, and refocused standards and 
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assessment requirements at state and district levels which became imbedded into Goals 
2000 and the 1994 Improving America’s Schools Act (IAS A). 

While the impact of the Council is difficult, if not impossible to measure and 
attribute even indirectly to CRESST, we do note that CRESST Co-directors Linn and 
Baker and other CRESST researchers are still consulted on a regular basis to advise the 
Department of Education on various aspects of Goals 2(XX), Title I of the Improving 
America’s Schools Act, and other aspects of legislation and policy related to standards 
and assessments. In 1995, CRESST staff participated in the mle making for lASA, 
particular to the use of assessments. In 1996, CRESST leadership participated in 
presentations across the country to provide information to public agencies about Title I 
standards and assessment requirements. Perceived as unbiased experts who are valued 
for their expertise and ideas, the influence of CRESST researchers appears to be 
considerable, with ^)parent impact on the standards and assessments of all fifty states 
and over 15,(XX) school districts nationally. 

Another example of CRESST impact at the national level is represented in their 
work for the U.S. Department of Education’s National Center for Education Statistics 
(NCES). Over the past five years NCES funded CRESST to convene a technical review 
panel to conduct validity studies of the National Assessment of Educational Progress 
(NAEP), oftentimes referred to as the nation’s report card. Scores firom the NAEP are 
reported across the country as an indicator of whether or not student achievement is 
increasing, decreasing, or staying the same. CRESST paitners conducted technical 
reviews of NAEP plans and documents, provided technical and policy advice to the 
National Assessment Governing Board which sets NAEP priorities, and conducted a 
series of independent studies of NAEP test design, administration, analysis, and 
reporting procedures. The results from the studies have been used to strengthen or 
improve the NAEP or in other cases, such as the setting of achievement levels, to 
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question the technical adequacy of certain procedures G-inn, Koretz, Baker, 1996). In 
addition, the Natiorud Academy of Education was commissioned to conduct an analysis 
of the NAEP trial state assessment. Leadership roles in this work were taken by 
CRESST researchers Robert Linn, Robert Glaser, and Lorrie Shepard. In addition, in 
their section on the redesign of NAEP, CRESST researchers Richard Shavelson and 
Eva Baker contributed chapters. 

Representative of the signifi cant impact on the national measurement community 
was the appointment of CRESST staff to the Joint Committee on Revising the 
Standards for Educatiorud and Psychological Testing. Co-sponsored by the American 
Psychological Association, the American Educational Research Association, and the 
National Council on Measurement in Education, the committee is in the midst of a 
comprehensive review and revision of the 1985 Standards. While the impact can oidy 
be estimated at this time, the revisions will almost certaiidy incorporate some of the new 
conceptual ideas related to validity and fairness as developed during the past 1 1 years of 
CRESST funding. Additionally, the revisions will reflect much of the CRESST agenda 
on appropriate uses of test results. CRESST researchers serve on the Board of Testing 
and Assessment of the National Research Council including Richard Shavelson, Richard 
Durdn and Robert Lirm. This group also sponsors workshops and reports on testing 
and assessment topics and numerous CRESST researchers including Robert Glaser, 
Daniel Koretz, Lorraine McDormell, and Harold O’Neil, Jr., have contributed to the 
work of this organization. 

State Impact As evidence of a state level impact basis, we cite the 1995 
report, “Review of the Measurement Quality of the Kentucky Instructional Results 
Information System, 1991-1994.” Although not a CRESST study, CRESST researchers 
Daniel Koretz and Robert Linn were co-authors of this report evaluating the technical 
quality of w hat was probably the most widely publicized implementation of a high- 
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stakes state performance assessment system in the United States. In the report, the 
researchers recommended that results from the Kentucky “portfolios not be used at this 
timft for accountability purposes; that validation work be expanded; that the performance 
standards be re-established and fiiU documentation of the process provided; that public 
reports be clear about limits of generalizability of findings to a new set of tasks; that 
multiple choice items be added to increase content validity and scoring reliability, and 
that the state reconsider its shift toward instructional process at the expense of 
curriculum content (McDonneU, in press). According to McDonneU, the Kentucky State 
Department of Education accepted a majority of the report findings and have since taken 
steps to implement many of the report’s recommendations. The impact of the report is 
considerable, in that the case can be made that the results from the study not only caused 
Kentucky to significantly change their assessment system, but that other states made 
major decisions about their own state assessment programs based on this highly 
publicized example. 

Other cases for CRESST-funded impact on state-level assessment programs 
could be made for Vermont, Arizona, Oregon, Washington, New York, Hawaii, North 
Carolina, Michigan, and Missouri, where CRESST has either conducted research or 
served as a consultant to state departments of education. In each state, CRESST 
research appears to have influenced major directions for state testing programs, 
frequently in terms of not pursuing certain strategies based on technical considerations 
or considering more dependable alternatives that were still aligned with instructional 
improvement. 

Another indicator of CRESST state level impact as weU as school district intact 
from CRESST research comes from the House evaluation (1996) “The Impact of 
CRESST R&D.” One element of the evaluation was to survey CRESST impact on state 
and local test directors, focusing on whether CRESST had influenced test director 
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decisions or their thinking about the use of alternative assessments. Based on a survey 
of more than 100 test directors, the evaluators concluded: 

When test directors were asked how useful CRESST was in building awareness 
of alternative assessments for testing directors, the majority (84%) thought that 
CRESST has been “useful” in generating awareness about these assessments. 
The majority (83%) of test directors also believed that CRESST was “useful” in 
di sseminating information about their function. Test directors also thought that 
CRESST was useful in influencing their thinking about alternative assessments 
(76%). Most (62%) believed that CRESST was useful in helping them make 
decisions about testing materials. 

In the same survey, verbal comments from the test directors generally praised 
both the quality of CRESST research and its dissemination: 

The literature published by CRESST influenced my thinking about alternative 
assessment which tmdoubtedly affected my decisions. 

Materials produced by CRESST have a sound research base and are objectively 
presented, unlike a lot of alternative assessment resources... The credibility of 
CRESST persormel and the widespread dissemination of CRESST publications 
have been useful. 

CRESST has been a major teacher in awareness; CRESST has been the leader 
nationall y in practical research on alternative assessment. The quality of their 
work is outstanding. 
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CRESST has consistently been on the “cutting edge” and is especially proficient 
in working with practitioners about what is going on in the “real world” of 
schools. 

It lets me know Fm not along out here. It’s evidence I can use with the power 
brokers. 

...The materials available through CRESST have been very helpful in helping us 
identify and think about the issues. 

In a tracer study from the same evaluation, the evaluators measured the intact of 
CRESST products, one of them a book by CRESST researchers Joan Herman, Pam 
Aschbacher, and Lyrm Winters.^^ Published by the Association for Supervision and 
Curriculum Development, “A Practical Guide to Alternative Assessment” was published 
in 1992 and distributed free to 90,000 AS CD members in addition to 43,650 sales in its 
first two and half years. Approximately 6600 copies were purchased by the Illinois 
Department State Board of Education for use in its regional training centers and 23,000 
copies were distributed to Chicago Public Schools. The House evaluation noted: 

Illinois public school admini strators found the book “quite helpful” in the state’s 
school improvement plarming process and use the book in training workshops 
given every six weeks for school staff. This individual recommends the book for 
workshops and uses its checklists for overiieads and presentations. All of the 
state directors and school improvement people have received copies of the book. 



^ Winters is now with the Long Beach Unified School District. 
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and it is referenced in the state’s publications. It has served as a “perfect fit” for 
the assessment component of the state’s school improvement plan, said a 
Chicago official because it helps teachers focus on the assessment of outcomes 
as something students can do or know, rather than as activities. 

While the preceding example represents pedraps the strongest impact on a state 
or school district program from this product, it was indicative, according to the 
evaluators, of “A Practical Guide to Alternative Assessment’s” wide acceptance and its 
ability to reach teachers. 

Local. CRESST influence at national and state levels subsequently impacts 
local school districts and indirectly, individual schools, oftentimes as a result of national 
legislation tied to fimding such as Goals 2000 or Title I or at individual state levels 
where CRESST has had direct input. CRESST is also involved in the development of a 
new assessment system for the Los Angeles Unified School District, the second largest 
school district in the nation, which if successful, could lead to much wider 
implementation in other districts and states. 
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Evaluation Question 3. What is the perceived quality and usefulness 
of CRESST research and products? How satisfled are CRESST 
customers? 



General D iscussion. 

Information across all three CRESST questionnaires was used to measure 
customer perceptions of CRESST quality, usefulness, and satisfaction. We present 
results on quality and usefulness from the CRESST Descriptive Questionnaire, focusing 
on the more global items therein, then present a composite picture of CRESST customer 
satisfaction from several items related to customers’ likeliness to use CRESST resources 
again, likeliness to recommend to CRESST to others, and the types of problems 
typically reported, or not reported. 

Findings on Quality of CRESST Research and Dissemination Produces. 

As discussed in the methods chapter, nine items from the CRESST Descriptive 
Questionnaire (Appendix A) were used to measure customer perceptions of quality of 
both CRESST research and dissemination products. Table 13 provides a summary of 
means, trimmed means, medians, and standard errors for all nine items. The means, 
trimmed means, and median all appear to represent relatively high levels of perceived 
quality, although again, comparisons to other research institutions are not available. 
Trimmed means are a more robust estimator of central tendency than medians (Norusis, 
1994). Ratings are from an eight-point scale. 



Table 13 

Means and Trimmed Means for Ratings of Quality of CRESST Research 



Variable 


Mean 


Trimmed Mean 


SD 


n 


Overall R&D 


6.66 


6.71 


1.03 


694 


Products 


6.59 


6.64 


1.03 


645 


Joumals/Books 


6.69 


6.74 


1.03 


545 


Tech Reports 


6.57 


6.63 


1.07 


577 


Presentations 


6.49 


6.55 


1.13 


312 


WWW 


6.39 


6.46 


1.20 


427 


Topics 


6.34 


6.41 


1.22 


712 


Newsletters 


6.29 


6.35 


1.14 


680 


Media Products 


6.20 


6.27 


1.31 


241 



For this dissertation, we focus on findings from two of the nine items that are 
broader measures of perceived quality, “overall quality of CRESST research” and 
“overall quality of CRESST products.” Appendix D has comprehensive tables and 
figures for all nine items. 

The distribution observed in Figures 1 1 and 12 are typical of responses to each 
item measuring both quality of CRESST research and quality of CRESST products. 
When responding to the question “how would you rate the overall quality of CRESST 
research,” 87.6% of all respondents rated the overall quality of CRESST research in the 
top three c^gories.” A similar percent (87.8%) rated CRESST products in the top three 
categories for the question “how would you rate the overall quality of CRESST 
products.” 
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Figure 1 1 

Ratings of Overall Quality of CRESST Research 
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Figure 12 

Bar Chart for Ratings of Overall Quality of CRESST Products 
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The findings are very consistent throughout the other seven items as presented in 
Appendix D. The bar charts are skewed to the right (negatively skewed) and suggest 
substantial satisfaction with CRESST research and its specific products including 
journal articles and books by CRESST researchers, newsletters, Internet services, media 
products, coverage of important topics, technical reports, and presentations. A number 
of problems; however, tended to bring down ratings for both the CRESST media 
products (M=6.20) and Internet services (M=6.29). As many as 10% of Internet users 
reported difficulties downloading CRESST products from its Web site. These had 
previously been identified in occasional e-mail messages from customers and a few 
phone calls. However, in every case investigated, the problem was linked directly to the 
user, especiaUy to the America On-Line browser. Fortunately, as the portable document 
format (PDF) has become mote standardized throughout the Internet industry and 
Internet suppliers improve their browsers, problems are abating. The somewhat low 
rating for CRESST media products (M=6.20), appears related to some dissatisfaction 
with the Alternative Assessment in Practice database, a HyperCard and Internet resource 
list of providers of alternative assessments. Half a dozen written comments from the 
CRESST descriptive questionnaire said that the CRESST Alternative Assessments in 
Practice Hatahasi*. contains many out of date sources. The database needs either quick 
attention or a link to the one of the regional laboratories with a current assessment 
database. 

Usefulness of CRESST Research and Disseminatifln Producls 

An equal number of items (9) from the CRESST descriptive questionnaire were 
used to measure the perception of usefulness of CRESST research and dissemination 
products (See Appendix D for fiiU results.) As with responses to the items on quality. 
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the means, medians, and trimmed means on the usefulness of CRESST research and 
dissemination products appear to be consistently high. Seventy-seven percent of all 
respondents (n=739) answered in the top three categories to the question “overall, to 
what extent has CRESST provided you useful information" and 82.8% answered in the 
top three categories to the question “overall to what extent do you believe that CRESST 
has been useful to the education cormnunity (n=751)." Table 14 provides a sununary of 
means, trimmed means, medians, and standard deviations for all nine items measuring 
usefulness of CRESST research and dissemination products. Complete results are 
reported in Appendix D. Ratings are on an eight-point scale. 



Table 14 

Means and Trimmed Means for Ratings of Usefulness of CRESST Research 



Variable 


Mean 


Trimmed Mean 


SD 


n 


Community 


6.54 


6.65 


1.30 


708 


Useful Inf. 


6.25 


6.36 


1.42 


739 


Joumals/Books 


6.39 


6.48 


1.30 


547 


Tech Reports 


6.20 


6.27 


1.36 


577 


Presentations 


6.17 


6.24 


1.35 


269 


WWW 


6.39 


6.52 


1.48 


426 


CRESST Ideas 


6.48 


6.58 


1.31 


657 


Newsletters 


6.09 


6.17 


1.41 


670 


Media Products 


6.08 


6.17 


1.52 


217 
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Because they are broader questions related to usefulness, we present only the 
charts from the questions “overall, to what extent has CRESST provided you useful 
informationT’ (Figure 13) and “overall, to what extent do you believe that CRESST has 
been useful to the education community?” (Figure 14). Seventy-seven percent of 
respondents rated “extent CRESST provided them useful information” in one of the top 
three categories while 82.8% rated CRESST usefulness to the education community in 
the top three categories. 



Figure 13 

Bar Chart for Ratings of Extent CRESST Has Provided You Useful Information 
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Figure 14 

Bar Chart for Ratings of Beliefs that 
CRESST has been Useful to the Education Community 
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Findings on CRESST Customer Satisfaction . 

Both the CRESST Product Questionnaire and CRESST Web Questionnaire 
included items that provided a measure of customer satisfaction levels. They are based 
on: 1) likeliness to use CRESST research in the future, and 2) likeliness to recommend 
CRESST to others. 

CRESST Product Questionnaire. Over 86% of respondents to the 
CRESST Product Questionnaire (Table 15) replied that they were either likely or very 
likely to order CRESST products in the future and 95.5% said that they were either 
likely, very likely, or already had recommended CRESST to others (Table 16). Twenty- 
seven percent (27.3%) of all respondents had already recommended CRESST to others. 
Only one respondent said that they were “very unlikely” to order CRESST products in 
the future but a review of that record did not reveal any reason for their unlikeness to 



onler in the future and they indicated that they were “very likely” to neconunend 
CRESST to others. Of the five respondents who answered that they were “somewhat 
unlikely” to order CRESST products in the future, four said they were either “likely” or 
“very likely” to reconunend CRESST to others. 



Table 15 

How Likely to Order CRESST Products Again 



Value Label 


Frequency 


Percent 


Vahd Percent 


Cum Percent 


Very unlikely 


1 


1.5 


1.5 


1.5 


Somewhat unlikely 


8 


11.8 


12.3 


13.8 


likely 


30 


44.1 


46.2 


60.0 


Vay likely 


26 


38.2 


40.0 


100.0 


Missing 


3 


4.4 


Missing 




Total 


68 


100.0 


100.0 
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Table 16 



How Likely to Recommend CRESST to Others 



Value Label 


Frequency 


Percent 


Valid Percent 


Cum Percent 


Veiy unlikely 


0 


0 


0 


0 


Somewhat unlikely 


3 


4.4 


4.5 


4.5 


Likely 


20 


29.4 


30.3 


34.8 


Veiy likely 


25 


36.8 


37.9 


72.7 


Already recomm. 


18 


26.5 


27.3 


100.0 


Missing 


2 


2.9 


Missing 




Total 


68 


100.0 


100.0 





CRESST Web Questionnaire. Similarly for the CRESST web, 88.5% of 
respondents said that they were either “likely” or “very likely” to use the CRESST web 
site in the future (Table 17) and 92.2% said that they were likely, very likely, or already 
had recommended the CRESST web to others (Table 18). No respondents said that 
they were unlikely to use the CRESST web again. Forty-four percent (44.2%) had 
already recommended the CRESST web site to others. 



Table 17 



Likeliness to Use CRESST Web Site in the Future 



Value Label 


Frequency 


Percent 


Valid Percent 


Cum Percent 


Very unlikely 


2 


3.7 


3.8 


3.8 


Somew unlikely 


4 


7.4 


7.7 


11.5 


Likely 


17 


31.5 


32.7 


44.2 


Very likely 


29 


53.7 


55.8 


100.0 


Missing 


2 


3.7 


Missing 




Total 


54 


100.0 


100.0 





Table 18 

How Likely to Recommend CRESST Web Site to Others 


Value Label 


Frequency 


Percent 


Valid Percent 


Cum Percent 


Very unlikely 


1 


1.9 


1.9 


1.9 


Somew unlikely 


3 


5.6 


5.8 


7.7 


Likely 


15 


27.8 


28.8 


36.5 


Very likely 


10 


18.5 


19.2 


55.8 


Already rec. 


23 


42.6 


M2 


100.0 


Missing 


2 


3.7 


Missing 




Total 


52 


100.0 


100.0 





A review of the questionnaire from the single web user who said they were 
“very unlikely” to recommend the CRESST web to others revealed that the respondent 
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was a parent associated with her school FTA who realized that CRESST did not provide 
the type of information she was interested in, that is, comparisons of student 
achievement in all school districts, public or private. She had also answered that she 
was “very unlikely" to use the CRESST web again. A review of the questionnaire from 
the second person who said that they were “very unlikely" to use the CRESST web 
again, showed that their primary interest was in evaluation methods and not K-12 
assessment which is the focus of CRESST research. 

Data to answer the question of customer satisfaction was also reviewed from an 
earlier 1995 pilot study (Dietel, 1995) which asked ten randomly selected telephone 
interview respondents “how likely" they were to order CRESST products in the future. 
Respondents were given four possible answers: very unlikely, somewhat likely, likely, 
and very likely to place future orders. Eight respondents said that they were “very 
likely" to order future products, one said “likely" and one said that “it depended on 
where their current project leads them." While only a low number of interviews were 
conducted, it does provide additional evidence to support the finding that a high 
percentage of users were satisfied customers. 

CRESST customer satisfaction was also measured by the number of problems 
reported with CRESST products or services based on open-ended question 25 firom the 
CRESST Descriptive Questioimaire, “what CRESST research was especially useful or 
problematicT’ Problems were coded into appropriate categories and reported in Figure 
15. 
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Figure 15 

Problems Reported with CRESST Research and Products 




What CRESST research or produa was problematic? 



Of 751 CRESST Descriptive Questionnaires returned, 57, or 7.6% reported 
something problematic while 230 of 751 respondents, 30. 6%, reported especially 
useful research or products. Of the problematic items, the largest group besides 
“Other,” was the CRESST web (28. 1%), with nearly all of these problems having 
trouble downloading portable document files which, as mentioned earlier, are seldom 
problems at the CRESST web site, but rather from the browser being used or the 
customers connection. By far the largest percentage of problems reported fell into the 
miscellaneous “other” category (40.4%), which was frequently inconsistent, oftentimes 
focusing on a single, very specific item that was not generalizable to other cases. A few 
examples follow on the next page: 
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• The AAIP didn’t have anything related to physics. I would like research 

on forming viable scientific models, overcoming misconceptions, strategies that 
contribute to the application of concepts. 

High school physics teacher 

• The PDF (acrobat) for mat is a pain in the neck. Much too slow to read or print. 
Most people use PCs not Macs.^^ 

School district senior evaluator 

• The video was problematic due to poor sound quality 

Assistant Principal 

• AAIP database had dated items like CLAS. Item banks for performance 
assessments would be useful as well as regional workshops. 

Manager of Evaluation and Research 

On the other hand, over 30% of all CRESST descriptive questionnaires returned 
indicated high levels of customer satisfaction with both CRESST research and 
dissemination products as shown in the examples that follow. Each of these also 
demonstrates some of the impact from CRESST research. 



^ The Acrobat Reader has always been available in both PC and Macintosh versions. 

^ Although this may not have been the same case, the one reported audio problem on a videotape was 
checked out and could not be found on a sample of other videotapes. The customer was offered a new 
videotape to replace the one with the reported problem. 





95 



• ‘Tracking Your School’s Success” and “A Practical Guide to Alternative 
Assessment” were very useful to us in working with schools and evaluation 
issues. 

Director of Evaluation, Wake City (NC) Public Schools 

• Title I information was especially usefiil. 

School District Test Director 

• The [CRESST] research on opportunity to learn was especially useful. The 
population I teach fits that category. 

Assistant Professor, Dade (Fla.) Cormnunity College 

• The newsletters keep me acquainted with current [assessment] undertakings. The 
technical reports have been especially useful for my research grants and 
portfolios, etc. 

Assistant Director of Office of Instructional Resources, Univ. of Florida. 

• I have particularly benefited from the timely discussion of important current 
issues in CRESST Line and Eval[uation] Comment. 

Director of Assessment, Washington Conunission on Student Learning 

• I used the CRESST web site to gather data for presentations to the Assistant 
Superintendent of Curriculum. I value the availability of technical reports/results 
of [CRESST] research projects. 

School district research assistant, Stockton (CA) Unified School District 

• Five years of CRESST research on CD ROM [was especially useful]. 

Secondary school English teacher, Assonet, Massachusetts 




96 




112 



• Assessing the Whole Child video [was especially useful]— provoked interesting 
and timely discussion about assessment issues. 

Co-director, Continuous Science Assessment Project (the Network, Inc.) 

• I read the newsletter because I admire the work of Eva Baker. I often share it 
with [the] Dean. . . 



• The reports on analytic scales and narrative writing by Gearhart, et al. were 
fabulous. 



Summary of findings on CRESST customer satisfaction. 

Overall, the short answer responses support other evidence presented in this 
section that CRESST is perceived as a valuable resource by a large majority of its 
customers and that the impact has been considerable in some cases. From the CRESST 
Descriptive Questiotmaire, over 70% of customers rated CRESST research and 
dissemination products in the top three categories of quality and usefulness on an eight- 
point scale. Customers indicate that they will use CRESST research in the future and 
many have already referred others to CRESST or report that they are likely or very likely 
to do so in the future. 



Assistant to Dean at Peabody College and former teacher 



Teacher from Spokane, Washington 
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Evaluation Question 4. Are there important differences across 
audiences between perceived quality and usefulness of CRESST 
research and dissemination products? 



General Discussion. 

For the CRESST Descriptive Questionnaire, it was hypothesized that market segments 
might exist between various independent variables, especially different registration lists, 
i.e., full mailing list, web list, or product order list; or different types of groups, i.e., 
school districts, K-12 schools, researchers, or others. CRESST products, for example, 
are not typically written for a practitioner audience, yet CRESST publications are sent to 
a substantial number of K-12 school practitioners (8.7%). It was surmised that ratings 
from K-12 audiences on the quality, or at minimum, the usefulness, of CRESST 
research, would be significantly lower than state or school district test directors or 
university researchers. Similarly, quality of the CRESST World Wide Web might be 
rated higher by those who had registered on the CRESST web vs. those who had 
ordered products or who were on the general CRESST mailing list. In order to detect 
important differences between means for the various groups discussed. Analysis of 
Variance was selected as the appropriate statistical procedure. However, because there 
was violation of some ANOVA assumptions, such as nonnoimality, nonparametric tests 
were used to determine the impact of violation of assumptions for the ANOVA results 
and QQ Plots were used to assess normality of the distributions (Appendix E). We 
discuss assumptions of normal vs. non-normal distributions, findings of differences in 
ratings of the quality and usefulness of CRESST and dissemination products across 
independent variables, both demographic and use-related, and the validity of the 
instrumentation used. 
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Assumption of Normal vs. Nonnormal distributions. The results from 
the QQ Plots indicated that the distribution for this study did not show significant 
departure from normality (Appendix E). The Levene test, used to analyze differences in 
population variances, provided evidence that except for a few cases the ANOVA 
assumption of equal variances was not violated. As discussed in the methods section, 
samples were all randomly generated using Filemaker Pro software, confirming the 
independence assumption necessary for ANOVA. The Bonferroni multiple comparison 
test was used to examine significant differences between key audiences, registration 
lists, and other independent variables, and the Scheff6 test, a more conservative estimate 
for pairwise comparisons (Norusis, 1994) was used for confirmation purposes. We 
discuss findings from the quality ratings first 

Findings about the differences in ratines of the quality of CRESST 
research and dissemination products. 

Gender. Of the nine items intended to measure quality of CRESST research 
and products, the only significant difference detected for the variable of gender was for 
the variable “quality of topics covered” [F(l, 703)=4.0355, p=.0449]. Therefore it 
seems safe to conclude that both nudes and females had similar perceptions regarding the 
quality of CRESST research and products. 

Race. The only significant difference found across race was for the dependent 
variable “overall coverage of important topics” with Caucasian (M=6.41) and Asian 
(M=5.63) samples showing a significant difference [F(S, 673)=4.873S, p<.01]. and 
Black (M^.72) and Asian (M=5.63) samples also significantly different. While the 
more restrictive Scheff6 test supported the findings from the Bonferroni, the Asian 
(n=30) and Black (n=23) sample sizes are so small as to prevent any strong inference 
from this finding. This interpretation is further supported because “overall coverage of 
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important topics” was the only dependent variable where race showed any significant 
difference. 

Group Audiences. For group audiences, the quality of journal articles/books 
was rated significantly different [F(3, 541)=4.17, p<.01] between group 1 
(districts/states, M=6.80) and group 4 (others, M=6.45); and group 3 (researchers, 
M=6.83) and group 4 (others, M=6.45), although the more conservative Scheffd test 
showed significance difference only between groups 3 and 4. There was a significant 
difference [F(3, 423)=2.81, p=.039] between group 1 (districts/states, M=6.54) and 
group 4 (others, M=6.07) for the dependent variable of quality of the CRESST World 
Wide Web although the Scheffd test did not support this finding. Because the “Other” 
group is rather undefined, this probably is not a very important finding. 

Regi.strations . Across types of registrants, that is across the three different 
mailing lists, significant differences were detected between group 1 (product list, 
M=6.53) and group 2 (web list, M=6.21) for the dependent variable “overall coverage 
of important topics” [F(2, 709)=4.85, p<.01]. The Scheffd test confirmed the 
significant difference. Only one other dependent variable from all nine quality 
dependent variables, “overall quality of CRESST research,” produced a sigmficant 
difference across registration type [F(2, 691)=4.21, p=.015], in this case, between 
group 1, product list customers (M=6.81) and group 3, the CRESST mailing list 
(M=6.54). However, because the significance is between two different lists, groups 1 
and 3, compared to groups 1 and 2 for the previous dependent variable, “quality of 
CRESST products,” any substantial inference of differences between registration types 
caimot be supported. 

Schools and Districts . The only significant difference detected between any 
schools or district independent variables was “type of school,” urban, rural, or 
suburban, for “quality of presentations...” [F(2, 46)=3.73, p=.03]. The significant 
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difference was found between urban (M=5.76) and suburban schools (M=6.68) and 
detected by both the Bonferroni and Scheff^ tests. 

When reviewed as a composite picture of CRESST users, the results suggest 
that at least for items measuring quality of CRESST research and products, CRESST 
customers are fairly homogeneous and market segmentation of products to specific 
groups of users would not be a cost-effective strategy. 

Other independent variables. Significant differences were found between 
responses to various quality dependent variables and many of the other independent 
variables i nr luding downloading PDF documents, ordered CRESST products, used the 
CRESST web, contact with CRESST staff, shared CRESST with others, and received 
newsletters. A summary table for one of the independent variables with the most 
significant differences, “shared CRESST with others," is provided in Table 19. 
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Table 19 



Significant Differences Between 

“Shared O^ST with Others” and Dependent Variables of Quality 

Never Once 2-3 times 4 or more 
DV (0) (1) (2) (3) df F p 



Overall 


6.13 


6.31 


6.56* 


Products 






(0) 


Presenta- 


6.50 


6.40 


6.29 


tions 








Important 


5.89 


6.00 


6.32* 


Topics 






(0) 


WWW 


5.77 


6.06 


6.36* 








(0) 


Jml/bks 


6.27 


6.29 


6.60 


Overall 


6.19 


6.36 


6.62* 


Research 






(0) 


News- 


6.01 


5.86 


6.29 


letters 








Media 


6.00 


5.70 


6.07 


Products 








Technical 


6.15 


6.19 


6.47 


Reports 









6.95* 

(0-2) 


3/629 


15.83 


<.01 


6.81* 

(2) 


3/304 


4.62 


<.01 


6 . 12 * 

(0-2) 


3/696 


14.44 


<.01 


6.75* 

(0-2) 


3/417 


10.15 


<.01 


7.07* 

(0-2) 


3/531 


15.94 


<.01 


7.05* 

(0-2) 


3/680 


20.37 


<.01 


6.65* 

(0-2) 


3/661 


16.13 


<.01 


6.50* 

(1) 


3/233 


6.46 


<.01 


6.97* 

(0-2) 


3/560 


16.99 


<.01 



• Indicates significant difference. Numbers in parentheses show subgroups that are 
significantly different to this category. 
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The pattern that emerged from the foregoing table is that a greater degree of 
usage across the independent variables of downloading PDF documents, ordering 
CRESST products, using the CRESST web, contact with CRESST staff, sharing 
CRESST with others, and to a lesser degree, receiving newsletters, leads to higher 
ratings of perceived quality. This is probably not a very surprising finding because 
increased levels of use probably reflect a better match between customer needs and 
CRESST as a research provider. What may be more important is that it does provide 
evidence that the CRESST Descriptive Questioimaire is sensitive to detecting different 
levels of use and corresponding ratings of quality. 

Findings about the difference s in ratines of the_usefulness of CRESSI 
research an d dissemination products. 

Gender, race, schoois, districts, and audiences. In general, ratings for 
usefulness of CRESST research and dissemination products followed very similar 
trends as those for quality, that is, significant differences were detected only across a 
few demographic independent variables. No significant findings were detected across 
gender, race, schools, or districts. Group audiences had only one significant difference 
[F(3, 543)=3.09, p=.027] and that was for journal articles between group 3 (researcher, 
M=6.52) and group 4 (other, M=6.04). Because it was hypothesized that group 
audiences would respond differently to the “overall, to what extent has CRESST 
provided you useful information” item, an independent samples t-test was performed to 
look at each sub-group, but none were significantly different. 

Registrations. Two significant differences were detected across mailing lists, 
group 1 (product order, M=6.26) and group 2 (web registrant, M=5.92) for usefulness 
of newsletters [F(2, 667)=3.63, p=.027], and between group 1 (product order, 
M=6.39) and group 3 (mailing list, M=5.97) for usefulness of technical reports [F(2, 
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574)=4.31, p=.014]. These few examples of significant differences between the three 
different mailing lists again suggest that the CRESST populations are fairly homogenous 
in their ratings of CRESST quality and usefulness. 

Other independent variables. As with ratings of quality, significant 
differences were found between many of the dependent ratings of usefulness and nearly 
all independent variables, nearly matching many of the observations from the quality 
ratings. For example, significant differences were found between “downloaded 
documents from the CRESST web” and seven of nine usefulness ratings (Table 20). 
Such a finding is not particularly surprising, although it does suggest that the CRESST 
Descriptive Questionnaire is effectively differentiating between levels of use, i.e. never 
downloaded PDF documents vs. downloaded PDF documents 4 times or more, and 
customer perceptions of usefulness. 





Table 20 

Differences Between 

“Downloaded Documents from the CRESST Web’”’ and 
Dependent Variables of Usefulness 



DV 


Never 

(0) 


Once 

(1) 


2-3 times 
(2) 


4 or more 
(3) 


df 


F 


P 


Educ. 


6.43 


5.96 


6.57* 


7.10* 


3/690 


12.83 


<.01 


Commun. 






(1) 


(0-2) 








Ideas from 


6.33 


6.13 


6.48 


6.95* 


3/636 


7.48 


<.01 


CRESST 








(0-2) 








WWW 


5.54 


5.73 


6.54 


7.23* 


3/414 


34.48 


<.01 








(0) 


(0-2) 








Jml/bks 


6.35 


6.02 


6.30 


6.74* 


3/531 


3.98 


<.01 










(0-1) 








Useful 


6.02 


5.84 


6.32 


7.02* 


3/715 


18.11 


<.01 


Information 






(0) 


(0-2) 








Media 


5.87 


5.67 


5.93 


6.61* 


3/207 


3.57 


.015 


Products 








(0) 








Technical 


6.08 


5.85 


6.10 


6.67* 


3/563 


6.75 


<.01 


Reports 








(0-2) 









* Indicates significant difference. Numbers in parentheses show subgroups that are 
significantly different to this category. 



Differences between similar items across constructs. What may be of 
greater interest was the finding that significant differences were found between the 
means of the quality and usefulness ratings for similar types of research or similar types 
of products including newsletters, technical reports, journal articles/books, and 
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presentations (Table 21). Note that for each of these instances, the quality rating is 
always higher, suggesting that some CRESST products and research may be perceived 
as having high quality, but having a lower usefulness level. 

Differences between quality and usefulness ratings for CRESST Internet 
services were not significant (Mq=6.39, M„=6.43, p=.463) nor were they significantly 
different between quality and usefulness ratings for media products (Mq=6.21, 
M„=6.10, p=.165). (See upcoming discussion on Factor Analysis.) 



Table 21 

Differences Between Dependent Variables Across Quality and Usefulness Constructs 



Value 


Mean 


SD 


df 


t 


P 


Newsl Quality 


6.31 


1.13 


660 


4.87 


<.01 


Newsl Useful 


6.10 


1.40 








TecRep (Quality 


6.61 


1.06 


544 


8.43 


<.01 


TecRep Useful 


6.22 


1.37 








Jml/Bks Quality 


6.73 


1.03 


501 


6.36 


<.01 


Jml/Bks Useful 


6.10 


1.40 








Present (Quality 


6.53 


1.13 


254 


4.87 


<.01 


Present Useful 


6.18 


1.36 








WWW Quality 


6.39 


1.20 


414 


-.73 


■463 


WWW Useful 


6.43 


1.45 








Media Quality 


6.21 


1.33 


204 


1.39 




Media Useful 


6.10 


1.51 
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Signiflcant differences between dissimilar dependent variables. 
Significant differences were found between dependent variables rating different items. 
For example, ratings of the quality of CRESST Internet services were significantly 
different than ratings of quality of CRESST journal articles [t(317)=S.lS, p<.01], and 
ratings for the quality of CRESST media products were significantly different than 
ratings for quality of CRESST technical reports [t(206)=6. 1 1 , p<.01 ]. 

The above findings suggest that there are clear differences between ratings of 
different types of CRESST dissemination products and provides additional evidence that 
the CRESST Descriptive Questionnaire can detect important differences when they exist. 

Validity of the CRESST Descri ptive Questionnaire. 

Reliability. As earlier stated, one of the purposes of this evaluation was to 
develop instrumentation that could dependably be used over the course of the current 
CRESST five-year grant and to potentially be shared with other research centers. Thus, 
validity of the CRESST Descriptive Questioimaire is important if dependable inferences 
are to be made from it. As a starting point, a variety of reliability tests were conducted 
across the nine items for each construct of the CRESST Descriptive Questioimaire. For 
items measuring quality, Cronbach’s alpha = .9370, Gutman split-half reliability 
coefficient = .9520, parallel maximum-likelihood reliability estimate =.9370 and strict 
parallel maximum-likelihood reliability estimate = .9295. For the usefulness items, 
Cronbach’s alpha = .8294, Gutman split-half reliability coefficient = .8650, parallel 
maximum-likelihood reliability estimate = .8924, strict parallel maximum-likelihood 
reliability estimate = .8774. These findings provide strong support for item reliability 
across the two constructs of quality and usefulness. 

Validity. As discussed earlier, the CRESST descriptive questionnaire did 
differentiate between ratings across a number of independent variables, including 
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sharing CRESST with others, personal contact with CRESST staff, use of the CRESST 
web, etc. We also found that the instrument detected significant differences between 
items measuring similar dissemination types across both quality and usefulness and 
found significant differences in all cases except media and Internet services. Finally, the 
instrument detected differences in ratings across different items and different constructs, 
supporting the overall validity of the instrument. 

Factor Analysis : Factor analysis was used to further analyze if the quality and 
usefulness items were measuring overall constructs of quality and usefulness. With all 
nine quality dependent variables entered into the analysis, only one factor was produced 
which provides additional evidence to support the ability of the items to appropriately 
measure the quality construct. For the usefulness construct, two factors were produced 
through the Varimax rotation (Table 22). Interestingly, the second factor contained the 
two items related primarily to technology, usefulness of the world wide web and 
usefulness of CRESST media products, with factor loading values of .82068 and 
.78991 respectively. Thus, it appears that the second usefulness construct could be 
called CRESST technology products. The other Factor 1 loadings from .69320 for 
newsletters to .86962 for journal articles/books seem to substantiate that these variables 
are measuring an overall construct of usefulness. 



Table 22 

Factor Analysis of Usefulness Items 
Varimax Rotated Factor Matrix 



Dependent Variable 


Factor 1 


Factor 2 


WWW usefulness 


.11662 


.82068 


Community usefulness 


.80139 


.28087 


Ideas usefulness 


.80567 


.06289 


Information to you useful 


.73864 


.42406 


Jml/Bks usefulness 


.86962 


.12928 


Media usefulness 


.25732 


.78891 


Newsletter usefulness 


.69320 


.12811 


Presentation usefulness 


.77074 


.29214 


TecRep usefulness 


.82245 


.32418 
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Evaluation Question 5. What else have we learned about the quality 
and usefulness of CRESST research and products? 



General Discussion. 

A number of other questions arose during the course of this evaluation that 
seemed important to measure for customer satisfaction purposes and program 
imp rnvp.mp.nt These questions were; how timely were CRESST products in reaching 
customers, what were the best methods to notify and deliver CRESST research products 
to customers, what time of year is the best or worst for notifying customers about the 
availability of CRESST products, and what type of information is sought and found on 
the CRESST web site? 

How timely were CRESST products in reaching customers? 

Specific data regarding the efficiency of CRESST product handling and 
shipment had not been investigated in recent years. Therefore, question 10 from the 
CRESST Product Questionnaire asked product recipients “How timely were the 
[CRESST] products in reaching you? Results are provided in Figure 16 (n=65). 

Almost 57% (56.9%) said they received their products in two weeks or less, 83. 1% of 
respondents said that they received products within one month of their order, 15.4% 
didn’t recall, and only one person reported having to wait more than two months for a 
product. No one reported that products took between 1-2 months to arrive. A review of 
the questionnaire from the person whose products took over two months to arrive did 
not provide additional information, however the customer said that they “were very 
likely to order CRESST products in the future.” 




no 



Figure 16 

Timeliness In Delivering CRESST Products 



n-65 




nro weefcB to one non don't recall 

How fiat products reached customers 



A review of the short answer responses to the more than 700 CRESST 
Descriptive Questioimaires returned revealed two comments where product orders were 
not prompdy completed and CRESST has since inq)lemented a procedure to notify 
customers if products will take more than one month to complete. The overall finding 
however, is that CRESST appears to fill orders promptly in nearly all circumstances. 

Best notification and receipt of CRESST products. CRESST 
dissemination has used a fairly broad approach to notifying customers of products 
including two newsletters, an annual product catalog, e-mail over a CRESST web 
registrant list, and the CRESST web site. The CRESST product questionnaire sought to 
determine if resoiuces might be better segmented into just one or two types of 
notification methods or if there was a preference for how products were delivered. 
Questions 1 1 and 12 on the CRESST Product Questionnaire, “what is the best way for 
you to receive notification of CRESST products?” and “what is the best way for you to 
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receive CRESST productsT’ investigated this issue. The results (Figure 17) suggest that 
CRESST needs to continue its present notification strategies because newsletters 
(34.0%), the product catalog (34.0%), and the CRESST web site (19.4%), all account 
for substantial levels of notification.” 



Figure 17 

Best way for customers to receive notification of CRESST Products 

1^65 






ProdiaCnbg CBESSTWcbSc Wcrdofmouh 

CieSCTVtib B-mrf 

BotmytoiccEivenadf atiaaof CRESSTprakcs 



Product Delivery. Findings also suggest that CRESST continue using 
mul tiple methods for product delivery. The best way for customers to receive products 
(Figure 18) still appears to be product orders (52.3%), followed by downloading 
CRESST research from the Internet (29.2%) and CD ROM (16.9%). Interestingly, 
while the Internet has greatly expanded CRESST dissemination, the preferred choice for 
product receipt is still via a product order. This may be due to lack of Internet access by 



” Because question 1 1 from the CRESST Product Questionnaire allowed two best ways for notification, 
the total number of responses coded was 103, although the sample size was 65. 
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many school districts who are the most frequent consumers of CRESST products. 
Similarly, while the CD ROM is a popular product, it has not replaced CRESST product 
orders as the piefened choice for product delivery. We should note that this was a 
sample from CRESST product orders only and therefore, may be biased towards that 
particular group. While many consumers ordering CRESST products have also 
accessed the CRESST Web (58.2%), any future studies should add a similar item to the 
CIRESST Web questionnaire, or alternately, a single item could be added to the 
CRESST Descriptive (^estionnaiie. 



Figure 18 

Best way for customers to receive CRESST Products 



n^5 




Doimload icscardi fr Any of 1-3 

Best way to receive CRESST products 



Best and worst time of year to order products. As mentioned earlier, 
anecdotal evidence and recent product sales data suggested that different times of the 
year are better for ordering CRESST products than others. Questions IS and 16 on the 
CRESST product questioimaiie asked ’Vhen is the time of year you are most likely to 
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Older CRESST products” and “when is the time of year you are least likely to order 
CRESST products?” Our findings did not show preferences from C31ESST product 
order customers (Figures 19 and 20). Almost eighty-six percent (85.9%) of respondents 
said that they did not have a “least likely” time of the year to order products confirmed 
by 87.7% who did not have a preference for “most likely” time of year to order 
products. (n=65 for both items.) The main concern was that newsletters or catalogs 
might be being mailed out at the wrong times of the year, or at minimum, that there 
might be optimum times for mailings. But this does not appear to be substantiated by 
the responses to the product questionnaire despite product sales records that show slow 
periods of orders during mid-late summer and December. 



Figure 19 

Least Likely Time of Year to Order CRESST Products 



n-65 



e. 




No diffocncc Summer Winter Spring Fall 



Time of year least likely to order CRESST products 




114 



Figure 20 

Most Likely Time of Year to Order CRESST Products 



n-65 




Best time of year to order CRESST products 

Additional questions related to the CRESST web site. Effective 
allocation of CRESST dissemination resources strongly suggests that CRESST direct 
the content of its world wide web to topics of greatest need, within the boundaries of the 
CRESST research program. Item 10 from the CRESST Web Questionnaire asked 
“what type of information have you looked for on the CRESST web?” Over 80% of 
respondents (n=55) marked more than one response, and many marked more than three 
or four responses. The results suggest that customers are looking for many different 
types of test related information on the CRESST web; however, when analyzed by 
individual counts, performance assessment information remained the most frequently 
sought out information (32.4%) followed by general information on testing (21%) and 
then portfolios (18.1%). (Figure 21.) 





115 



Figure 21 

Information Sought on the CRESST Web 



n-55 




Pcrf asst info Portfolios Tcm Ocher 



rVn Info on Stand Test Info Test Scores from sea 

Infonnadon sought on CRESST web 



Regarding the degree to which information sought is found on the CRESST web 
site, 15% of CRESST web registrants reported that they found nearly all (more than 
75%) of the information they sought, 41.5% reported that they found much (between 
51-75%) of the information sought, 32.1% reported that they found some of the 
information sought (between 10-50%) and 7.5% reported that they found almost none 
of the information sought (less than 10%). (Figure 22.) While comparative information 
is not available, this seems to suggest that CRESST web users are finding a sigmficant 
amniint of the information they seek on the CRESST web site. 
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Figure 22 

Success in Finding Desired Information on the CRESST Web Site 



n<*53 




Some (10*S0%) Afanott oobc (<10K) 

Success finding information sought on CRESST web 



o 
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Chapter 5 

DISCUSSION AND RECOMMENDATIONS 

Chapter Introduction. 

In this chapter, we discuss the findings from each evaluation question, 
limitations of this evaluation, and suggested areas for further research or evaluation. In 
general, our discussion emphasizes those particular questions where results have 
important implications for the present and future CRESST dissemination program. As 
stated earlier, our focus was primarily to evaluate the mass media strategies of the 
CRESST Dissemination Model (Figure 1) emphasizing one-way diffusion of important 
R&D information but with some crossover to information tools of the model and to a 
lesser degree, interactive dissemination. The results from this evaluation show that 
CRESST is most often an information provider to change agents who are likely to 
synthesize CRESST research with other available information and existing belief 
structures to induce reform of assessment and classroom instruction. 

Evaluation Oue.stion 1. What did we learn about the ex isting use of 
CRESST research and dissemination products2 

Audiences. The data from this section indicate that CRESST research use and 
mass media dissemination is fairly evenly spread across gender, targeted occupations, 
an d geographic regions. Gender usage tends to be evenly split, with a slight edge to 
females (53.4%) over males (46.6%) based on data from the CRESST mailing list. The 
race of CRESST consumers (Appendix D), measured by the Descriptive Survey, is 
heavily Caucasian (85%) followed by Asian/Pacific Islander (4.2%), Latino (3.8%), 
black (3.4%), and Native American (.7%). That the CRESST research audience 
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q>peais to be primarily Caucasian is reflective of the educational research field in 
general. 

The CRESST mailin g list shows more occupational variation than either the web 
registrant list or product order list, most likely reflecting the differences between a self- 
selection process for the latter two, and a list selectively chosen by CRESST 
management from various conference and mailing lists of perceived important 
assessment audiences. Interestingly, the web registrant list reflected a substantial K-12 
audience, about 23%, which is likely to grow as schools gain better Internet access. 

That state government employees, mostly state departments of education, represent only 
a s mall percentage of product orders (2.7%) seems a bit surprising, but it is possible that 
they receive assessment information from CRESST in other ways, perhaps personal 
contact with CRESST researchers, or that they receive assessment information from 
other sources such as the Council of Chief State School Officers. 

Discovery. CRESST research on its web is typically discovered through a 
general Internet search on education topics (28.8%) or a general Internet search on the 
topic of testing (21.2%). This findings suggests that at least 50% of CRESST web 
registrants have not previously heard of CRESST, and thus these customers represent a 
new outreach opportunity. Combined with other data, such as the downloading of 
CRESST PDF files, it is obvious that the CRESST web site is an integral and essential 
mass media for disseminating CRESST research, especially to new customers. 

Product Usage. That CRESST product sales have not fallen off more steeply 
as a result of making the same products available over the Internet and via CD ROM may 
be a result of limited Internet access by many primary customers, generally school 
districts, or it may be a result of a continued preference for traditional ordering systems. 
We speculate that CRESST product sales will remain flat, or more likely, gradually 
decrease, dependent on how many new products are made available to users. Less 
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administrative time is likely to be necessary for filling product orders, time which may 
be better transferred to other functions, especially maintenance or enhancement of the 
CRESST web site. 

A monthly statistical analysis capability to better track CRESST web statistics, 
should be a top priority for the CRESST dissemination program. Of particular interest 
would be possible trends suggested in this evaluation of longer connection times to the 
CRESST web site and a substantial increase in downloading of PDF files noted during 
the final period of 1996. 

Product Life Cycle: The analysis of life cycles across four CRESST 
products confirms that long-term demand exists for certain types of mass media 
products, especially those that have broad content ^peal, such as an assessment model 
handbook or videotape. Nevertheless, the low number of overall individual product 
sales, for any CRESST product, suggests that CRESST products have very specific 
audience appeal to somewhat small audiences, or that some improvements in marketing 
might increase usage. We have not discussed results from satellite broadcasts for 
CRESST videotapes nor have we considered that CRESST oftentimes provides free 
copies of reports to substantial numbers of researchers or consumers who are close 
collaborative parmers or in jobs, such as media, where CRESST wishes to build and 
establish long-lasting relationships. Nor do these cormnents include book sales or 
journal articles that are outside of CRESST distribution. Finally, we have not compared 
CRESST product sales to those from similar research centers which might show that 
CRESST sales are quite high in comparison. 

TCvaluation Oue.stion 2. How are CRESST research an d products used? 

Extent read, reviewed, and shared. Because of the limited time that many 
education consumers t 5 q>ically have to spend on research reading, our hypothesis was 
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that consumers who order CRESST products, regardless of good intentions, might only 
scan them, or not even look at them once in hand. Considering also the density of 
CRESST research and in many cases, their technical nature, it was somewhat surprising 
to find that nearly 90% of CRESST customers read or reviewed between 10-100% of 
the CRESST materials downloaded or purchased. As in nearly all parts of this study, 
comparative information would be extremely helpful. 

Equally surprising was the sharing of CRESST research with others with only 
12% of total audiences never sharing CRESST infoimation. Probably reflective of the 
ways in which CRESST research becomes transparent as it filters from one source to 
another, this finding at minimum confinns the hypothesis that CRESST research is 
shared on a broad basis. More importantly, this finding suggests that there are still 
substantial numbers of consumers who are not receiving CRESST research materials but 
who may be finding the information useful. How those consumers might be identified 
is not clear but we think that some marketing techniques might be useful if managemen t 
considers this an important expansion opportunity. 

Use and impact of CRESST research. Although our question from the 
CRESST Product Questionnaire provided some insight into actual use of CRESST 
research, that is, about 25% being shared with teachers or used as background 
information, and 13% to inform policy, the open-ended questions were much more 
specific and revealing. What seemed interesting was the diversity of responses, fiom 
CRESST research added to an informational database at the National Center on 
Educational Outcomes, to information from the CRESST web site being used by the 
Tennessee Department of Education director of evaluation and assessment to inf orm the 
development of the Tennessee state assessment program. We suggest that the next 
generation CRESST Descriptive Survey include an item or items directly related to 
impact, i.e., what was the impact of CRESST research? (See upcoming suggestions for 
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future study.) A model for research impact might well provide further categories 
beyond those presented in our findings: 1) impact on practice, 2) impact on policy with 
implications for practice, and 3) impact on measurement theory and research. At 
minimum, a comprehensive model would have categories for audiences, methods of 
impact, and multiple levels of impact. 

Evaluation Question 3. What is the perceived quality and^sefulness of 
CRESST research an d products? 

Our findin gs support the conclusion that, in general, CRESST research and 
mass media products are perceived as high quality and very useful (Tables 13 and 14). 
The major surprises in the ratings on quality and usefulness were the very high ratings 
for journal articles and books published by CRESST researchers, M=6.69 for quality 
and M=6.39 for usefulness. We were concerned in particular that some audiences 
would find books and journal articles by CRESST researchers a bit too academic and 
while perhaps of high quality, not quite so useful. But for both quality and usefulness, 
journals and books authored by CRESST researchers had the third highest mean rating 
of any dependent variable. 

The somewhat low means for both quality and usefulness of CRESST 
newsletters, M=6.29 for quality and M=6.09 for usefulness, placed them second to last 
on both constructs measured. This finding somewhat contradicts much anecdotal 
evidence that newsletters were very well received in the field, but the relatively large 
sample sizes, n=680 for quality of newsletters and n=670 for usefulness of newsletters, 
is strong evidence of consistency and suggests some strategies for improvement. 
Comments from the questiormaires were not especially revealing as to what might 
improve the newsletters, but we think, as with most research, that content is extremely 
imp ortant as well regular delivery which has not always been the case with the CRESST 
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newsletters. There were positive comments about the newsletters too, and we should be 
careful not to misinteipret what could be audience sentiments that, in general, 
newsletters are an overused media, lack specific substance or data, or focus too closely 
on news or individual achievements that have little relevancy for research consumers. 

CRESST Customer Satisfaction: Customer satisfaction ratings were 
constructed based on two questions, likeliness to use CRESST research in the future 
and likeliness to recommend CRESST to others. In retrospect, a more direct question, 
i.e., how satisfied are you as a CRESST research consumer, might be a more effective 
measure of customer satisfaction. Nevertheless, customer satisfaction appears to be 
substantial with well over 80% of respondents intending to “use CRESST research in 
the future" and similar percentages for “likeliness to recommend CRESST to others" or 
having already done so. Again, comparative information would be very helpful. There 
are a few specific product problems that need resolution, in particular, the Alternative 
Assessment in Practice database. CRESST must weigh the fact that the database is one 
of its most highly used products, based on web statistical usage data, against the 
resources and time it will take to produce an update. One possible alternative is to link 
the AAIP database to a similar one maintained by one of the regional educational 
laboratories. 

Evaluation Question 4. Are there important differences across audiences 
between perceived quality and usefulness of CRESST research and 
dissemination products? 

Our findings from the CRESST Descriptive Questionnaire provided good 
evidence that the populations as they presently exist on the three CRESST mailings lists 
and across various subgroups, are quite homogenous. Even if differences were not 
detected on ratings of CRESST research and mass media product quality, we at least 
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expected to detect more differences than found in the ratings of usefulness. As a 
research provider with limited resources for either research or dissemination, the results 
may be a pleasant surprise, as needs to segment markets appear unwarranted. Future 
evaluations of CRESST dissemination may use the results from this study to argue 
against large sample sizes as long as random samples stratify well across CRESST 
audiences. 

How important significant differences are across various CRESST mass media 
products, that is, newsletters, web site, journal articles/books, etc., is really a CRESST 
management decision. It seems that as long as quality and usefulness ratings do not 
drop significantly from existing levels, that no specific action is required, except as 
earlier discussed about newsletters and the AAIP database. 

5. What else have we learned abo ut the quality and usefulness of 
CRESST research and products? 

The questions contained in this section, 1) timeliness of CRESST products in 
reaching customers, 2) best notification and receipt of CRESST products, 3) best and 
worst time of year to order CRESST products, and 4) most often sought information on 
the web site and to what degree found, are well presented in the Chapter 4, Findings. 
Our comments are therefore brief. 

Overall, CRESST products were delivered to customers in a timely maimer with 
only a few instan ces of delay. It seems relatively unlikely that CRESST could do much 
to improve their product delivery system although they may want to add the convenience 
of credit cards for payment. CRESST is advised to keep their existing notification and 
delivery methods for CRESST mass media products since customers are using them in 
si gnifi cant numbers and express a desire to continue doing so in the future. Since 
customers generally expressed no seasonal preference for ordering products, CRESST 
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need not adjust their notification media to anticipate specific good or bad ordering 
periods, although they may wish to avoid mid- to-late summer and December when 
possible. It seems likely that most CRESST product order requests are not urgent since 
approximately one-third or more products are ordered via a purchase order which is 
typically not a quick method for obtaining a product. Of course the CRESST web is an 
excellent media for innmftrfiatfi access to the latest CRESST research and we can only 
envision its increasingly important role in the future. 

Our results confirm the continued popularity of performance assessments as the 
type of information most frequently sought on the CRESST web site. The fact that 
CRESST has conducted more recent research on this topic than anyone else in the 
country and makes it downloadable at no cost from its web site, may help to account for 
what seems to be a relatively high success rate in customers finding the research sought. 

We conclude this section by mentioning a single comment from an open-ended 
questiormaire item. A customer conq>lained about the slippery nature of the CRESST 
technical report covers and while it may seem to be a small issue, the covers have also 
been noted by CRESST persormel to slip around on the shelves in the CRESST 
publications room. New covers in development should try to avoid similar problems 
that result from high gloss finishes on publications. 

Limitations of this evaluation. 

This evaluation was limited by the lack of conq)arative information from similar 
research institutions or from previous evaluations where baseline data was available; by 
the lack of reliable and valid instrumentation from other studies that could be adapted to 
this evaluation, and by the relatively low sample sizes produced for both the product and 
web questioimaires. 
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Comparative Information: While the purpose of this evaluation was 
formative, the results would have much greater impact and utility if comparisons to other 
organizations were available. Summative evaluation, for example, strongly emphasizes 
the idea of comparing the value or worth of something in comparison to something else 
(Scriven 1967; 1973b). Sharing the instruments from this evaluation with other 
research centers or any of the ten regional educational laboratories may provide the type 
of comparative information that would be helpful. A presentation made at the October, 
1996 CEDaR communicators group resulted in at least one laboratory expressing an 
interest in using the CRESST instruments. Regardless of the outcome, the results from 
this evaluation will provide baseline data and instrumentation that should be useful over 
the course of the five-year CRESST research center evaluation and beyond. 

Ins trumentation and records. The creation of new instruments for any 
research study or evaluation leads quickly to issues of reliability and validity. Although 
initial evidence of reliability and validity for the CRESST Descriptive Questionnaire has 
been presented, similar information was not collected for the CRESST Product or Web 
Questionnaires. Also, the statistical tracking software used to analyze the CRESST web 
usage, although state of the art, may still lead to inaccurate estimates given that web 
usage is estimated from the number of connections made to a server. CRESST product 
records, although carefully tracked, are sold in such low numbers as to decrease their 
relevancy unless reviewed over a minimum three-four month period of sales. Despite 
these limitati ons, the overall consistency of the results^ suggests that instrumentation and 
records were dependable and the mferences made from data collected should be quite 
trustworthy. 
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Suegestions for further study 



Although this study focused on the quality and usefulness of CRESST research 
and its mass media products, it did produce evidence of significant impact that was 
occurring in the field. Nevertheless, the accurate measurement of impact from 
educational research remains elusive. On a macro level, one possible next step is the 
convening of a small focus group of dissemination specialists and center directors or 
associate directors from other research centers, together with representatives from their 
funding institution, the Office of Educational Research and Improvement, to discuss: 1) 
reasonable expectations for what mq)act research centers and laboratories should be able 
to accomplish in the course of their five-year programs, 2) specific methods across all 
research centers to measure many diverse forms and levels of impact, and 3) realistic 
possibilities for sharing conq)arative information in a non-competitive environment. 

On a micro level, reasonable next steps at CRESST could be the adaptation of 
the CRESST Descriptive Questionnaire to focus on measuring and reporting research 
impact. Such an instrument should include situations where research has prevented 
possible misuse of assessment as well as those cases where research has been integrated 
into practice, policy, or other research. A revised CRESST Descriptive Questionnaire 
could reduce the total number of questions asked if it was not essential to differentiate 
between products. CRESST might also want to consider the design of an instmment that 
could be used across all CRESST research projects as ongoing measurement of impact. 

Summary of important findings and implications for CRESSTLi-esearch 
and dissemination. 

We briefly summarize the major findings from this study of the dissemination 
program of the National Center for Research on Evaluation, Standards, and Students 
Testing (CRESST): 
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1) The quality and usefulness of CRESST research appears to be high. Between 
70%-90% of consumers rated the quality and usefulness of CRESST research in 
the top three categories of performance on an eight-point scale. Some rethinking 
ma y be reco mmende d for the content of CRESST newsletters with the goal of 
increased ratings of quality and usefulness. A fast-back survey soliciting 
preferred newsletter topics might be helpful. 

2) Based on the finding that, in general, different demographic categories of 
CRESST consumers gave similar ratings to the quality and usefulness of 
CRESST research, we conclude that CRESST audiences are relatively 
homogeneous and that market segmentation of products and research does not 
seem appropriate at this time. Overall customer satisfaction appeared to be high 
and most consumers perceived CRESST as a valuable and trusted resource. 

3) The CRESST web site has more than tripled the number of products now 
reaching consumers and has become an essential part of the CRESST 
dissemination program. Because of its relatively low operating cost, 24-hour a 
day availability, ability to deliver comprehensive research almost immediately, 
and no cost to consumers, we expect usage to increase. Improved monitoring of 
usage rates would be useful and administrative resources might eventually be 
shifted from har d copy product orders to the web site. With increased Internet 
access to schools and other educational organizations, the Internet probably 
represents the best opportunity, and perhaps the only cost-effective opportunity, 
for CRESST research to reach a vast and mostly untapped market. 

4) Impact from CRESST research is considerable in all current CRESST 
dissemination strategies, that is, mass media, interactive dissemination, and 
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information tools. EspeciaUy impressive has been the impact on national, state, 
and local poUcy makers which either has had, or has the potential for, significant 
impact on practice. Because of limited resources, CRESST might better focus 
on products with wide dissemination oppoitunities such as the book, “A 
Practical Guide to Alternative Assessment," which has had widespread impact, 
especiaUy on practice. Current strategies that attempt to edit and market eveiy 
possible product, might be more efficiently focused on fewer, selective 
products. Performance assessment, portfoUos, and assessment models or 
assessments themselves, remain popular topics. 

5) Comparative information from other research centers or similar organizations 
would aid decision makers in measuring program quality, usefulness, impact, 
and needed areas of improvement. Refinement of instruments from this study 
could be adapted to: 1) better document CRESST impact; 2) answer new 
questions in place of those that may not need to be revisited, i.e. best/worst time 
of year to purchase CRESST products; and 3) for sharing with other centers and 
laboratories. 

6) Although the purpose of this evaluation was to focus on the mass media 
strategies of the CRESST dissemination model, a comprehensive evaluation 
must focus on aU dissemination strategies in the model, especiaUy interactive. 
We think that it is not too early to plan interactive dissemination strategies and 
evaluation of major new CRESST programs, including the CRESST/LAUSD 
coUaboration (Assessment and Instmction Models), QuaUty Schools Portfolios, 
and QuaUty Education Forum, among others. 
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APPENDIX A: 



CRESST Descriptive Questionnaire 



CRESST DESCRIPTIVE QUESTIONNAIRE 



Name 

Organization 

Job Titie (piease be specific) 

Address 

Phone (business)^ 

If at schooi district, approximateiy how many students are in your district? 

If at school, approximately how many students are at the school? 

Is the school □elementary, □ middle, or □ secondary? □ not applicable 

Is the district or school primarily □ urban, □ rural, or □ suburban? □ not applicable 



Aecea« to CRESST Begourees and Products 





within the past two years, how often Unsure 
have you... 


Never 


Once 


2-3 times 


4 or more 
times 


1. 


had Darsonal contact with any CRESST Staff?' 


[ I 


[ I 


[ I 


[ I 


[ I 


2. 


...received anv CRESST newsletters? 
(CRESST Une/Evaluatlon Comment) 


[ I 


[ I 


[ I 


[ I 


[ I 


3. 


...ordaied CRESST oroducts? 

(technical reports, videotapes, CD ROMs) 


[ I 


[ I 


[ I 


[ I 


[ I 


4. 


—used the CRESST Web ^www.cse.uc/aedu)|7 


[ I 


[ I 


[ I 


[ I 


[ I 


5. 


. . .downloaded documents from the CRESST Web? 


[ I 


[ I 


[ I 


[ I 


[ I 


6. 


—shared CRESST research or products with others? 


[ I 


[ I 


[ I 


[ I 


[ 1 



Quality of CRESST Resource s and Products 
Quality Includes the professional nature of the research and appearance of the product compared 



How would you rate the... 


Never 

ueed Poor 


Fair 


Good 


Excellent 


7 - flualf*y of CRESST newsletters? 


/ J 1 2 


3 4 


5 6 


7 8 



' CRESST Staff Includes research and administrative staff. CRESST sponsored 
researchers are: Eva Baker, Robert Unn, Jamal Abedi, Pamela Aschbacher, Hilda Borko, R. Darnell 
Bock, Lee J. Cronbach, Richard DurSn, Robert Glaser, Edmund W. Gordon, Joan Herman, Daniel 
Koretz, Karen Mitchell, Lorraine McDonnell, Robert Mislevy, Bengt Muthdn, Harold O’Neil, Jr., Lauren 
Resnick, Richard Shavelson, Lorrie Shepard, Richard Snow, Ronald Stevens, and Noreen Webb. 
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8. quality of CRESS! media productsZ [ ] 

(videos, CD ROMs, Alternative 
Assessment In Practice database) 

9. quality of CRESS! technical reports? [ ] 

10. quality of journal articles or books [ ] 

written by CRESS! researchers?'* 

11. quality of presentations bv 

CRESS! researchers? [ ] 



Questionnaire Nunnber. 



1 


2 


3 


4 


5 


6 


7 


8 


1 


2 


3 


4 


5 


6 


7 


8 


1 


2 


3 


4 


5 


6 


7 


8 


1 


2 


3 


4 


5 


6 


7 


8 








CONTINUED 


ON BACK 



Quality of CRESS! Resources and P roducts (continued) 



Never 

How would you rate the... used 


Poor 


Fair 


Good 


Excellent 


12. quality of CRESST Internet services? 
(CRESST Web site: www.cse.uda.edu) 


[ 1 


1 


2 


3 


4 


5 


6 


7 


8 


1 a nynraii quality trf CRESST research? 
(content, researchers, methods) 


1 J 


1 


2 


3 


4 


5 


6 


7 


8 


1 a overall quality of CRESST products? 
(content, appearance, professionalism) 


[ ] 


1 


2 


3 


4 


5 


6 


7 


8 


1 5. overall coverage of important tooics? 


[ ] 


1 


2 


3 


4 


5 


6 


7 


8 


Usefulness of 


CRESST 


Resources 











Usefulness includes the application of a CRESST product or sen/ice to meet your or your organization s 
needs. 



How useful are. 



17. CRESSTtechnical reports? 

18. |oumal articles or books written bv 
CRESS! researchers?'* 



Never Seldom 
used Useful 



] 
[ ] 
[ ] 



1 



2 

2 

2 



Sometimes 

Useful 



3 

3 

3 



4 

4 

4 



Fairly 

Useful 



5 

5 

5 



6 

6 

6 



Very 

Useful 

7 8 

7 8 

7 8 



19. CRESST newsletters? 



[ ] 1 



8 



20. CRESST Internet services? [ ] 1 

(CRESST Web sHe: www.cse.ucla.edu) 

21. CRESST media products? [ ] 1 

(videos, CD ROM, Alternative 

Assessment in Practice Database) 



2 3 4 5 6 7 



8 



O 

FRir 
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22. Ideas generated bv CRESST? 11 '^ 2 3 4 5 

(Validity criteria, technical Issues of 

portfolios and performance 
assessments, Title I guidance) 

23. Overall, to what extent has CRESST [ ] ‘^ 2 3 4 5 

provided you useful information? 

24. Overall, to what extent do you believe [ ] ^ 2 3 4 5 

that CRESST fws been useful t o the 

education community? 

25. What CRESST research or product was especialiy useful or problematic? Why?. 



26. What new CRESST research, product, or sendee would be useful to you?. 



Beekground Questions 

Race: □Black; □ Caucasian; □ Latino; □ Asian/Pacific Islander; □ Native American; □ 

Other 

Gender □ Female GMale 

May we call you for a few additional questions? □ Yes □ No 



APPENDIX B: 

CRESST Product Questionnaire 
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CRESST Product Questionnaire 



Name 

Organization 

Job Title (please be specific) 

AeeesB to CRESST Resources and Products 
Within the past two yean, how often Unsure Never Once 2-3 times 4 or more 





have you... 








times 


1. 


...had oersonal contact with anv CRESST Staff?' 


[ ] 


[ ] 


[] [ ] 


[ ] 


2. 


...ordered CRESST oroducts? 


[ ] 


[ ] 


[] [ ] 


[ ] 




(technical reports, videotapes, CD ROMs) 










3. 


...used the CRESST Vfeto (www.cse.uda.edu)? 


[ ] 


[ ] 


[ ] [ ] 


[ ] 



4. ... shared CRESST research or products with others?f 1 [] [] [] [] 

Questions 

5. Your interest in CRESST is primarily as a . 

6. How did you find out about the CRESST products you ordered? 

□ CRESST Publication (□ CRESST Line; □ Evaluation Comment; □ Product 

Catalog) 

□ Printed mention in non-CRESST publication 

□ Colleague or Professor 

□ Other 

□ Don’t recall 



' CRESST Staff includes research and administrative staff. CRESST affiliated 
researchers include: Eva Baker, Robert Linn, Jamal Abedi, Pamela Aschbacher, Richard DurSn, 
Robert Glaser, Edmund W. Gordon, Joan Herman, Daniel Koretz, Karen Mitchell, Lorraine McDonnell, 
Robert Mislevy, Bengt Muth4n, Harold O’Neil, Jr., Lauren Resnick, Richard Shavelson, Lorrie 
Shepard, Richard Snow, Ronald Stevens, and Noreen Webb. 





7. What was the primary purpose of your order? (Mark top one or two purposes only.) 

□ Research 

□ Student Research 

□ Test Development 

□ Inform Policy 

□ Infomi Teachers 

□ Inform Students 

□ Other 

8. To what extent did you use the products you ordered? 

□ Read, reviewed, or used less than 10% of the materials 

□ Read, reviewed, or used about 10-50% of the materials 

□ Read, reviewed, or used about 50-75% of the materials 

□ Read, reviewed, or used over 75% of the materials 

9. What did you do with the information you viewed or retrieved? 



1 0. How timely were the products in reaching you? 

□ Received products less than two weeks after placing order 

□ Received products between two weeks and one month after placing order 

□ Received between one and two months after placing order 

□ Received more than two months after placing order 

□ Don’t recall 

11. What is the best way for you to receive notification of CRESST products? (Number the two 
best.) 

□ Word of mouth (CRESST researchers or colleagues) 

□ Direct Mailing of Product Catalog 

□ CRESST Newsletters (CRESST Line or Evaluation Comment) 

□ E-mail 

□ The CRESST Web site 

□ Other 

12. What is the best way for you to receive CRESST products? (Seiect one only.) 

□ Product Order 

□ Annual CD ROM with CRESST products from previous year 

□ Downloadable research from the CRESST Web 

□ Other 
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1 3. How likely are you to order CRESST products in the future? 



Very 


Somewhat 


Likely 


Very 




unlikely 


unlikely 




likely 




[] 


[] 


[] 


[] 




How likely are you to recommend CRESST to others? 


Very 


Somewhat 


Likely 


Very 


Already 


unlikely 


unlikely 




likely 


recommended 










to others 


[] 


[] 


[] 


[] 


[] 



15. When is the time of year you are most likely to order CRESST products? 

Fan Winter Spring Summer No difference 

[] [] [] [] [] 

16. When is the time of year you are least likely to order CRESST products? 

Fall Winter Spring Summer No difference 

[] [] [] [] [] 

1 7. What can we do to improve our products or our ability to inform you of their 

availability? 
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APPENDIX C: 



CRESST Web Questionnaire 



CRESST Web Questionnaire (www.cse.ucla.edu) 



Name 

Organization 

Job Title (please be specific) 

AeeoBB to CRESST Resourceg and Products 

Within the past two years, how often Unsure 
have you... 

1. — had personal contact with any CRESST Staff?’ [ ] 

2. ... ordered CRESST products? [ ] 

(technical reports, videotapes, CD ROMs) 

3. .. .used the CRESST Web ^iwm.cse.octe.ecM7 [] 

4. — Shared CRESST research or products with others? [] 

5. Your Interest in CRESST is primariiy as a 

6. How did you find out about the CRESST Web Server? 

□ Colleague or Professor recommendation 

□ Printed Mention in CRESST Publication (□ CRESST Line. □ Evaluation Comment. 

□ Product Catalog) 

□ Printed Mention in non-CRESST Publication: (name of publication) 

□ Mention or reference from other Web Site 

□ General internet Search for Education Topics 

□ General Internet Search for Assessment or Testing Topics 

□ Other^ 



Never 


Once 


2-3 times 4orm< 
times 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 


[ ] 



' CRESST Staff includes research and administrative staff. CRESST affiliated 
researchers include: Eva Baker. Robert Linn. Jamal Abedi. Pamela Aschbacher. Richard Durdn. 
Robert Glaser. Edmund W. Gordon. Joan Herman. Daniel Koretz. Karen Mitchell. Loneine McDonnell. 
Robert Mislevy. Bengt Muthdn. Harold O’Neil. Jr.. Lauren Resnick. Richard Shavelson. Lorrie 
Shepard. Richard Snow. Ronald Stevens, and Noreen Webb. 
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7. What type of Internet access do you have? (Select best one only.) 

□ University □ American On-line □ Telnet 

□ Compuserv □ Prodigy □ Other 

□ School □ Other commercial provider □ Don’t Know 

8. How often do you currently visit the CRESST site? (Select best one only.) 

Almost A few times Almost Almost Almost Not 

never every six months monthly weekly Daily Sure 

n n II n n n 

9. In the past six months, about how many reports or other documents have you 
downloaded from the CRESST web site?^ 

None or Unsure About 1 -3 About 3-5 About 5-1 0 About 1 0-20 More than 20 



n [] [] n n n 



10. What type of information have you looked for on the CRESST Web? {Mark any that 

appiyi 



□ General information on testing 

□ Performance-based assessment information 

□ Standardized norm-referenced test information 

□ Tests 



□ Test Scores from States/Districts 

□ Portfolios 

□ Other 

□ Don't recall 



1 1 . To what extent have you found the information you hoped to find? 

□ Almost none of the information hoped for (less than 10%) 

□ Some of the information hoped for (from 10-50%) 

□ Much of the information hoped for (from 51-75%) 

□ Nearly all of the information hoped for (more than 75%) 

□ Don't recall 

12. What specific information do you recall viewing on the CRESST web site? {Mark any that 
apply.) 

□ Alternative Assessments in Practice Database 

□ Newsletters (either CRESST Line or Evaluation Comment- PDF Files) 

□ General hterest Papers (including resource papers) 

□ Video Summaries 



2 



CRESST uses the Adobe Acrobat Reader in the portable document format for downloading large files. 



□ CRESST Technical Report Sumniaries 

□ CRESST Technical Reports (PDF Files) 

□ Education and Assessment Links to other Web sites 

□ Doni recall 

13. To what extent did you use any materials viewed or retrieved? 

□ Read, reviewed, or used less than 10% of the materials 

□ Read, reviewed, or used about 10-50% of the materials 

□ Read, reviewed, or used about 50%-75% of the materials 

□ Read, reviewed, or used over 75% of the materials 

□ Don’t recall 

14. What did you do with the information you viewed or retrieved? 



15. What information or feature could be added to the CRESST web site to make it more 
useful to you? 



16. How likely are you to use the CRESST web site in the future? 

Very Somewhat Likely Very 

unlikely unlikely likely 

[] [] [] N 

17. How likely are you to recommend the CRESST web site to others? 

Very Somewhat Likely Very Already 

unlikely unlikely likely recommended 

to others 

[] [] [] [] [] 
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APPENDIX D 



Descriptive Statistics and Charts from CRESST Descriptive Survey 



Chart 1 

Cross Tab Chart of Counts for T vpe of Registration 

bv Group^BAJdiences 



GR0U_REG Type of Registration by GR0U_AUD IV Type of Subgroup 

GR0U_AUD 



Count I 

Tot Pet I District School Research Other 
I /State er 

I 1| 2| 3| 



Row 

5 I Total 



GROU_REG 

Product 

Internet 

Full Rolodex 



Coluxnn 

Total 



81 

10.8 


1 

1 


46 

6.1 


1 

1 


79 

10.5 


1 

1 


49 

6.5 


1 

1 


255 

34.0 


71 

9.5 


1 

1 


76 

10.1 


1 

1 


87 

11.6 


1 

1 


61 

8.1 


1 

1 


295 

39.3 


53 

7.1 


1 

1 


43 

5.7 


1 

1 


66 

8.8 


1 

1 


39 

5.2 


1 

1 


201 

26.8 


205 




165 




232 




149 




751 



27.3 



22.0 



30.9 



19.8 



100.0 




143 

159 



Descript ive Statistics aod Chart-2. 
Gender 



Valid Cum 



Value Label Value 

Female ^ 
Male 2 
Omitted 99 



Total 



Frequency 


Percent 


Percent 


Percent 


381 


50.7 


51.3 


51.3 


361 


48.1 


48.7 


100.0 


9 


1.2 


Missing 




751 


100.0 


100.0 





60 
so 

40 

30 
20 
10 
P« 0 

rv 1 -Female 2-Male 



GENDER IV 


1-Female 


2 -Male 






Valid 


Cum 


Value Label 




Value 


Frequency 


Percent 


Percent 


Percent 


Female 




1 


381 


50.7 


51.3 


51.3 


Male 




2 


361 


48.1 


48.7 


100.0 


Omitted 




99 


9 


1.2 


Missing 








Total 


751 


100.0 


100.0 




Mean 


1.487 


Std err 


.018 


Median 


1.000 


Mode 


1.000 


Std dev 


.500 


Variance 


.250 


Kurtosis 


-2.002 


S E Kurt 


.179 


Skewness 


.054 


S E Skew 


.090 


Range 


1.000 


Minimum 


1.000 


Maximum 


2.000 


Sum 


1103.000 








Valid cases 


742 


Missing 


cases 9 











Female Male 
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Descriptive Statistics and Chart 3 
Race 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Black 


1 


24 


3.2 


3.4 


3.4 


Caucasian 


2 


605 


80.6 


84.9 


88.2 


Latino 


3 


27 


3.6 


3.8 


92.0 


Asian/Pac 


4 


30 


4.0 


4.2 


96.2 


Native Azn 


5 


5 


.7 


.7 


96.9 


Other 


6 


22 


2.9 


3.1 


100.0 


Omitted 


99 


38 


5.1 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




C«nca«i«n AsUn/Ptc Other 



Mean 


2.233 


Std err .032 


Median 


2.000 


Mode 


2.000 


Std dev .862 


Variance 


.743 


Kurtosis 


10.013 


S E Kurt .183 


Skewness 


3.092 


S E Skew 


.092 


Range 5.000 


Minimum 


1.000 


Maximum 


6.000 


Sum 1592.000 






Valid cases 


713 


Missing cases 38 







145 




Descriptive Stati stics and Chart 
Type of Re gistration 



Value Label Value 

Product 1 
Internet 2 
Full Rolodex 3 



Total 







Valid 


Cum 


Frequency 


Percent 


Percent 


Percent 


255 


34.0 


34.0 


34.0 


295 


39.3 


39.3 


73.2 


201 


26.8 


26.8 


100.0 


751 


100.0 


100.0 





I 



Mean 


1.928 


Std err .028 


Median 


Mode 


2.000 


Std dev .776 


Variance 


Kurtosis 


-1.331 


S E Kurt .178 


Skewness 


S E Skew 


.089 


Range 2.000 


Minimum 


Maximum 


3.000 


Sum 1448.000 




Valid cases 


751 


Missing cases 0 





Bar Chart 




Produa Intcnct FuU Bolod ei 



2.000 

.603 

.125 

1.000 
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Descriptive Stat istics and Chart 5 
Type of Group 



Value Label 



Value Frequency Percent 



Valid Cum 
Percent Percent 



District/State 1 
School 2 
Researcher 3 
Other 5 



205 


27.3 


27.3 


27.3 


165 


22.0 


22.0 


49.3 


232 


30.9 


30.9 


80.2 


149 


19.8 


19.8 


100.0 



Total 



751 100.0 100.0 



Bar Chart 




Dutrin/SciK School Bdcvchcr Other 



Mean 


2.631 


Std err 


Mode 


3.000 


Std dev 


Kurt os is 


-.878 


S E Kurt 


S E Skew 


.089 


Range 


Maximum 


5.000 


Siam 



. 051 Median 3 . 000 

1.404 Variance 1.972 

.178 Skewness .512 

4 . 000 Minimiom 1 . 000 

1976.000 



Valid cases 751 Missing cases 0 
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Descriptive Statistics and Cha rt 6 





District 


size 




Valid 


Cucci 


Value Label 


Value Frequency 


Percent 


Percent 


Percent 


Small District 


1 


113 


15.0 


44.0 


44.0 


Mediiun District 


2 


67 


8.9 


26.1 


70.0 


Large District 


3 


77 


10.3 


30.0 


100.0 


NA 


0 


480 


63.9 


Missing 




Omitted 


99 


14 


1.9 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Small District Medium Dimict Large Distria 



Mean 


1.860 


Mode 


1.000 


Kurtosis 


-1.565 


S E Skew 


.152 


Maximum 


3.000 


Valid cases 


257 



Std err .053 
Std dev .850 
S E Kurt .303 
Range 2.000 
S\m 478.000 

Missing cases 494 



Median 2.000 
Variance .722 
Skevmess .272 
Minimiun 1.000 



O 

ERIC 
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Descriptive Statistics and Chart 7 
Type of District 



flu 



Valid Cum 

Value Label 




Value 


Frequency 


Percent 


Percent 


Urban 




1 


107 


14.2 


40.5 


Rural 




2 


53 


7.1 


20.1 


Suburban 




3 


93 


12.4 


35.2 


More than one 




4 


11 


1.5 


4.2 


NA 




0 


462 


61.5 


Missing 


Missing 




99 


25 


3.3 


Missing 






Total 


751 


100.0 


100.0 


Hi -Res Chart 


# 78: Bar 


chart of iv 


type of district 




Mean 


2.030 


Std err 


.059 


Median 


Mode 


1.000 


Std dev 


.963 


Variance 


Kurtosis 


-1.366 


S E Kurt 


.299 


Skewness 


S E Skew 


.150 


Range 


3.000 


Minimum 


Maximum 


4.000 


Sum 


536.000 






Valid cases 


264 


Missing cases 487 







Bar Chart 





149 



Percent 

40.5 

60.6 
95.8 

100.0 



2.000 

.927 

.223 

1.000 



Descript ive Statistics and Chart _£ 
Type of School 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Elementary 


1 


67 


8.9 


39.9 


39.9 


Middle 


2 


20 


2.7 


11.9 


51.8 


Secondary 


3 


60 


8.0 


35.7 


87.5 


Multiple 


55 


21 


2.8 


12.5 


100.0 


NA 


0 


568 


75.6 


Missing 




Omitted 


99 


15 


2.0 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Meeui 
Mode 
Kurtosis 
S E Skew 
Maximum 


8.583 

1.000 

3.246 

.187 

55.000 


Std err 
Std dev 
S E Kurt 
Range 
Sum 


1.359 

17.618 

.373 

54.000 

1442.000 


Median 

Variance 

Skewness 

Minimum 


2.000 

310.388 

2.277 

1.000 


Valid cases 


168 


Missing 


cases 583 








Descriptive Stat istics and Chart 9 
Size of Schools 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Small School 


1 


93 


12.4 


57.8 


57.8 


Medium School 


2 


48 


6.4 


29.8 


87.6 


Large School 


3 


19 


2.5 


11.8 


99.4 


Multiple schools 


4 


1 


.1 


.6 


100.0 


Wi 


0 


568 


75.6 


Missing 




Omitted 


99 


22 


2.9 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 

7 0 




Small Scfaocil Lai^c School 

Mcdhun School Multiple achools 



Mean 


1.553 


Std err 


.057 


Median 


1.000 


Mode 


1.000 


Std dev 


.724 


Variance 


.524 


Kiirtosis 


.014 


S E Kurt 


.380 


Skewness 


1.016 


S E Skew 


.191 


Range 


3.000 


Minimum 


1.000 


Maximum 


4.000 


Sum 


250.000 







Valid cases 161 



Missing cases 590 
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Descript ive Statistics and Chart lil 
Locatio n of School 











Valid 


Cvun 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Urban 


1 


49 


6.5 


29.7 


29.7 


Rural 


2 


34 


4.5 


20.6 


50.3 


Suburban 


3 


75 


10.0 


45.5 


95.8 


More than one 


4 


7 


.9 


4.2 


100.0 


NA 


0 


566 


75.4 


Missing 




Omitted 


99 


20 


2.7 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Mean 

Mode 

Kurtosis 
S E Skew 
Maximtim 


2.242 

3.000 
-1.305 

.189 

4.000 


Std err .073 
Std dev .932 
S E Kurt .376 
Range 3.000 
Siam 370.000 


Median 

Variance 

Skewness 

Minimvun 


2.000 

.868 

-.182 

1.000 


Valid cases 


165 


Missing cases 586 







ERIC 
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Descriptive Statistics and Chart 11 
Contact with CRESST Staff 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Never 


0 


388 


51.7 


54.3 


54.3 


Once 


1 


111 


14.8 


15.5 


69.8 


2-3 Times 


2 


139 


18.5 


19.4 


89.2 


4 or more times 


3 


77 


10.3 


10.8 


100.0 


Unsure 


9 


28 


3.7 


Missing 




Omitted 


99 


8 


1.1 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Nevtr 


Once 


2-3 Times 4 or more dmcs 








Me£in 


.867 


Std err 


.040 


Median 


.000 


Mode 


.000 


Std dev 


1.073 


Variance 


1.152 


Kurtosis 


-.839 


S E Kurt 


.183 


Skewness 


.791 


S E Skew 


.091 


Range 


3.000 


Minimum 


.000 


Maximum 


3.000 


Sum 


620.000 






Valid cases 


715 


Missing cases 36 
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Descriptive Statistics and Char t ..12 
Received Newsletters 



Valid 



Value Label 


Value 


Frequency 


Percent 


Percent 


Never 


0 


54 


7.2 


7.4 


Once 


1 


71 


9.5 


9.7 


2-3 times 


2 


257 


34.2 


35.3 


4 or more times 


3 


347 


46.2 


47.6 


Unsure 


9 


18 


2.4 


Missing 


Omitted 


99 


4 


.5 


Missing 




Total 


751 


100.0 


100.0 



Bar Chart 




Never 


Once 


2‘3 dmci 4 or more times 




Meeui 


2.230 


Std err .033 


Median 


Mode 


3.000 


Std dev .904 


Variance 


Kurt os is 


.347 


S E Kurt .181 


Skewness 


S E Skew 


.091 


Range 3.000 


Minimum 


McLximum 


3.000 


Sum 1626.000 




Valid cases 


729 


Missing cases 22 
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Cum 

Percent 

7.4 

17.1 

52.4 

100.0 



2.000 

.818 

-1.074 

.000 



Descriptive Statistics and Chart 13 
Ordered CRESST Products 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Never 


0 


301 


40.1 


42.5 


42.5 


Once 


1 


202 


26.9 


28.5 


70.9 


2-3 times 


2 


164 


21.8 


23.1 


94.1 


4 or more times 


3 


42 


5.6 


5.9 


100.0 


Unsure 


9 


33 


4.4 


Missing 




Omitted 


99 


9 


1.2 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Never 


Once 


2-3 rimci 4 or more cfanci 






Mean 


.925 


Std err .035 


Median 


1.000 


Mode 


.000 


Std dev .943 


Variance 


.888 


Kurtosis 


-.815 


S E Kurt .183 


Skewness 


.576 


S E Skew 


.092 


Range 3.000 


Minimum 


.000 


Maximum 


3.000 


Sum 656.000 






Valid cases 


709 


Missing cases 42 
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Descriptive Statistics and Chayt M 
used CRESST Web 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Never 


0 


300 


39.9 


40.5 


40.5 


Once 


1 


63 


8.4 


8.5 


49.1 


2-3 times 


2 


143 


19.0 


19.3 


68.4 


4 or more times 


3 


234 


31.2 


31.6 


100.0 


Unsure 


9 


5 


.7 


Missing 




Omitted 


99 


6 


.8 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Mean 
Mode 
Kurtosis 
S E Skew 
Maximum 


1.420 

.000 

-1.723 

.090 

3.000 


Std err 
Std dev 
S E Kurt 
Range 
Sum 


.048 

1.300 

.179 

3.000 

1051.000 


Medicin 

Varicince 

Skewness 

Minimum 


2.000 

1.689 

.053 

.000 


Valid cases 


740 


Missing 


cases 11 
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Descriptive Statistics and Chart 15 
Downloaded PDF Dociiments 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


never 


0 


392 


52.2 


53.6 


53.6 


Once 


1 


61 


8.1 


8.3 


62.0 


2-3 times 


2 


155 


20.6 


21.2 


83.2 


4 times or more 


3 


123 


16.4 


16.8 


100.0 


Unsure 


9 


9 


1.2 


Missing 




Omitted 


99 


11 


1.5 


Missing 






Total 


751 


100.0 


100.0 





Hi -Res Chart # 86: Bar chart of iv downloaded pdf documents 



Bar Chart 
6 0 1 — 



so> 



40> 



30* 



20 * 



5 lO- 

s 



0 , 



never 


Once 


2-3 limci 4 ciinet or man 








Mean 


1.012 


Std err 


.044 


Median 


.000 


Mode 


.000 


Std dev 


1.193 


Variance 


1.423 


Kurtosis 


-1.322 


S E Kurt 


.181 


Skewness 


.573 


S E Skew 


.090 


Range 


3.000 


Minimum 


.000 


Maximum 


3.000 


Sum 


740.000 






Valid cases 


731 


Missing cases 20 
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Descriptive Stati stics and Chart -1^ 
Shared CRESST with Others 











Valid 


C\m 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Never 


0 


87 


11.6 


11.8 


11.8 


Once 


1 


107 


14.2 


14.6 


26.4 


2-3 times 


2 


320 


42.6 


43.5 


69.9 


4 or more times 


3 


221 


29.4 


30.1 


100.0 


Unsure 


9 


12 


1.6 


Missing 




Omitted 


99 


4 


.5 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Mean 


1.918 


Std err 


Mode 


2.000 


Std dev 


Kurtosis 


-.454 


S E Kurt 


S E Skew 


.090 


Range 


McLximum 


3.000 


Svm 


Valid cases 


735 


Missing 



.035 Median 2.000 

.956 Variance .914 

.180 Skewness -.652 

3.000 Minimum .000 

1410.000 

es 16 





Descriptive Statistics and Chart 17 
Quality of CR ESST Newsletters 



Value Label Value 

Poor Plus 2 

Fair Minus 3 

Fair Plus 4 

Good Minus 5 

Good Plus 6 

Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

Unsure 9 

99 



Total 







Valid 


C\un 


Frequency 


Percent 


Percent 


Percent 


2 


.3 


.3 


.3 


8 


1.1 


1.2 


1.5 


27 


3.6 


4.0 


5.4 


107 


14.2 


15.7 


21.2 


257 


34.2 


37.8 


59.0 


165 


22.0 


24.3 


83.2 


114 


15.2 


16.8 


100.0 


66 


8.8 


Missing 




1 


.1 


Missing 




4 


.5 


Missing 




751 


100.0 


100.0 





Bar Chart 

4 0 




Poor Plus Fair Plus Good Plus Eicdlent Plus 

Fair Minus Good Minus R»fgiient Minus 



Mean 
Mode 
Kurtosis 
S E Skew 
Maximum 


6.294 

6.000 

.232 

.094 

8.000 


Std err 
Std dev 
S E Kurt 
Range 
Svm 


.044 

1.139 

.187 

6.000 

4280.000 


Median 

Variance 

Skewness 

Minimum 


6.000 

1.298 

-.384 

2.000 


Valid cases 


680 


Missing 


cases 71 
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Descriptive Statistics and Chart 1£ 
Quality of CRESST Media Products 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Fair Minus 


3 


11 


1.5 


4.6 


4.6 


Fair Plus 


4 


13 


1.7 


5.4 


10.0 


Good Minus 


5 


38 


5.1 


15.8 


25.7 


Good Plus 


6 


79 


10.5 


32.8 


58.5 


Excellent Minus 


7 


56 


7.5 


23.2 


81.7 


Excellent Plus 


8 


44 


5.9 


18.3 


100.0 


Omitted 


0 


496 


66.0 


Missing 




Unsure 


9 


2 


.3 


Missing 






99 


12 


1.6 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Fair Minus Good Minus Exoeilcnc Minus 



Fair Plus Good Phis ExceUcnt Plus 



Mean 

Mode 

Kurtosis 
S E Skew 
Maximum 


6.195 

6.000 

-.113 

.157 

8.000 


Std err 
Std dev 
S E Kurt 
Range 
Slim 


.084 

1.310 

.312 

5.000 

1493.000 


Median 

Variance 

Skewness 

Minimum 


6.000 

1.716 

-.512 

3.000 


Valid cases 


241 


Missing 


cases 510 
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Descriptive Stat istics and Chart 19 
Quality of CRESST Technical Reports 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Fair Minus 


3 


7 


.9 


1.2 


1.2 


Fair Plus 


4 


13 


1.7 


2.3 


3.5 


Good Minus 


5 


61 


8.1 


10.6 


14.0 


Good Plus 


6 


176 


23.4 


30.5 


44.5 


Excellent Minus 


7 


201 


26.8 


34.8 


79.4 


Excellent Plus 


8 


119 


15.8 


20.6 


100.0 


Omitted 


0 


166 


22.1 


Missing 




Unsure 


9 


1 


.1 


Missing 






99 


7 


.9 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Fair Mmui Good Mkun EaccUcnt Mimii 



Fair Ptua Good Plus ExceUcm Ptiu 



Mean 


6.574 


Std err 


.045 


Median 


7.000 


Mode 


7.000 


Std dev 


1.073 


Variance 


1.151 


Kurtosis 


.432 


S E Kurt 


.203 


Skewness 


-.636 


S E Skew 


.102 


Range 


5.000 


Minimum 


3.000 


Maximum 


8.000 


Sum 


3793.000 






Valid cases 


577 


Missing 


cases 174 








1 
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Descriptive Statistics and Chart 20 
Quality of Journal Articl es /Books Authored bv CRfiSSE 

Researchers 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Fair Minus 


3 


5 


.7 


.9 


.9 


Fair Plus 


4 


7 


.9 


1.3 


2.2 


Good Minus 


5 


47 


6.3 


8.6 


10.8 


Good Plus 


6 


163 


21.7 


29.9 


40.7 


Excellent Minus 


7 


193 


25.7 


35.4 


76.1 


Excellent Plus 


8 


130 


17.3 


23.9 


100.0 


Omitted 


0 


196 


26.1 


Missing 




Unsure 


9 


1 


.1 


Missing 






99 


9 


1.2 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




flu Oi 


Fair Minua 


Good Min 111 


Excdleni Minua 












Fair Plu» 


Good Plua Excdleni 


Plu> 






Meeui 




6.692 


Std err 


.044 


Median 


7.000 


Mode 




7.000 


Std dev 


1.026 


Variance 


1.052 


Kurt OS is 


.513 


S E Kurt 


.209 


Skewness 


-.638 


S E Skew 


.105 


Range 


5.000 


Minimum 


3.000 


Maximum 


8.000 


Sum 


3647.000 






Valid 


cases 


545 


Missing cases 206 
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O 

ERIC 
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Descriptive Statistics and Chart 21 
Quality of CRESST Prese ntations 











Valid 


Ciun 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Fair Minus 


3 


1 


.1 


.3 


.3 


Fair Plus 


4 


17 


2.3 


5.4 


5.8 


Good Minus 


5 


40 


5.3 


12.8 


18.6 


Good Plus 


6 


83 


11.1 


26.6 


45.2 


Excellent Minus 


7 


111 


14.8 


35.6 


80.8 


Excellent Plus 


8 


60 


8.0 


19.2 


100.0 


Omitted 


0 


422 


56.2 


Missing 




Unsure 


9 


3 


.4 


Missing 






99 


14 


1.9 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 

40i 




Fair Minus Good Minus Excellent Minus 

Fair Plus Good Plus Excell ent Plot 



Mean 


6.494 


Std err 


.064 


Median 


7.000 


Mode 


7.000 


Std dev 


1.123 


Variance 


1.260 


Kurt os is 


-.292 


S E Kurt 


.275 


Skewness 


-.526 


S E Skew 


.138 


Range 


5.000 


Minimum 


3.000 


Maximum 


8.000 


Sum 


2026.000 






Valid cases 


312 


Missing 


cases 439 
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Descripti ve Statistics and Chart _2_2 
Quality of CRESST Internet Servic.^ 



Value Label Value 

Poor Minus 1 

Poor Plus 2 

Fair Minus 3 

Fair Plus ^ 

Good Minus 3 

Good Plus ^ 

Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

Unsure 8 

99 



Total 







Valid 


Ciun 


Frequency 


Percent 


Percent 


Percent 


1 


.1 


.2 


.2 


2 


.3 


.5 


.7 


6 


.8 


1.4 


2.1 


16 


2.1 


3.7 


5.9 


56 


7.5 


13.1 


19.0 


142 


18.9 


33.3 


52.2 


124 


16;5 


29.0 


81.3 


80 


10.7 


18.7 


100.0 


312 


41.5 


Missing 




1 


.1 


Missing 




11 


1.5 


Missing 




751 


100.0 


100.0 





Bar Chart 




Poor Minus 


Fair Plus 


Good Plus Excellent 


Plus 








Fair Minus Good Minus Excciient Minus 










Mean 


6.386 


Std err 




.058 


Median 


6.000 


Mode 


6.000 


Std dev 




1.198 


Variance 


1.435 


Kurtosis 


1.211 


S E Kurt 




.236 


Skewness 


-.780 


S E Skew 


.118 


Range 




7.000 


Minimum 


1.000 


Maximiim 


8.000 


Sum 


2727.000 






Valid cases 


427 


Missing cases 


324 









164 



180 



Descript ive Statistics and Chart 23 
Overall Quality of CRES ST Research 



Value Label Value 

Poor Minus 1 

Fair Minus 3 

Fair Plus 4 

Good Minus 5 

Good Plus 6 

Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

99 



Total 







Valid 


Cum 


Frequency 


Percent 


Percent 


Percent 


1 


.1 


.1 


.1 


2 


.3 


.3 


.4 


14 


1.9 


2.0 


2.4 


69 


9.2 


9.9 


12.4 


193 


25.7 


27.8 


40.2 


264 


35.2 


38.0 


78.2 


151 


20.1 


21.8 


100.0 


46 


6.1 


Missing 




11 


1.5 


Missing 




751 


100.0 


100.0 





Bar Chart 
2001 




Fair Minus Good Minus Eioellcnt Minus 



Fair Plus Good Plus ExccUcnt Plus 



Mean 


6.660 


Std err 


.039 


Median 


7.000 


Mode 


7.000 


Std dev 


1.029 


Variance 


1.059 


Kurtosis 


1.060 


S E Kurt 


.185 


Skewness 


-.718 


S E Skew 


.093 


Rctnge 


7.000 


Minimum 


1.000 


Maximum 


8.000 


Sum 


4622.000 







Valid cases 694 Missing cases 57 
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Descriptive ^Stati stics and Chart 2A 
Quality of CRESST Product g 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Fair Minus 


3 


3 


.4 


.5 


.5 


Fair Plus 


4 


20 


2.7 


3.1 


3.6 


Good Minus 


5 


56 


7.5 


8.7 


12.2 


Good Plus 


6 


211 


28.1 


32.7 


45.0 


Excellent Minus 


7 


223 


29.7 


34.6 


79.5 


Excellent Plus 


8 


132 


17.6 


20.5 


100.0 


Omitted 


0 


94 


12.5 


Missing 




Unsure 


9 


2 


.3 


Missing 






99 


10 


1.3 


Missing 





Total 751 100.0 100.0 



Bar Chart 




Fair Minus Good Minus Excellent Minus 



Fair Plus Good Fhu Excellent Plus 



Mean 


6.592 


Std err 


.041 


Median 


7.000 


Mode 


7.000 


Std dev 


1.033 


Variance 


1.068 


Kurtosis 


.165 


S E Kurt 


.192 


Skewness 


-.528 


S E Skew 


.096 


Range 


5.000 


Minimum 


3.000 


Maximum 


8.000 


Sum 


4252.000 






Valid cases 


645 


Missing 


cases 106 







O 
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Descript ive Statistics and Chart 25 
Quality of T opics Covered 



Value Label Value 

Poor Minus 1 

Poor Plus 2 

Fair Minus 3 

Fair Plus 4 

Good Minus 5 

Good Plus 6 

Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

Unsure 9 

99 







Valid 


Cum 


Frequency 


Percent 


Percent 


Percent 


1 


.1 


.1 


.1 


2 


.3 


.3 


.4 


12 


1.6 


1.7 


2.1 


38 


5.1 


5.3 


7.4 


95 


12.6 


13.3 


20.8 


222 


29.6 


31.2 


52.0 


221 


29.4 


31.0 


83.0 


121 


16.1 


17.0 


100.0 


16 


2.1 


Missing 




3 


.4 


Missing 




20 


2.7 


Missing 





Total 751 100.0 100.0 



Bar Chart 




Foot Pltif Fair Plui Good Flua EaccUent Plui 



Fair Minita GoodMinua EzoeUcm Minus 



Heaxi 


6.341 


Std err 


.045 


Median 


6.000 


Mode 


6.000 


Std dev 


1.206 


Variance 


1.454 


Kurtosis 


.628 


S E Kurt 


.183 


Skewness 


-.703 


S E Skew 


.092 


Range 


7.000 


Minimum 


1.000 


Maximum 


8.000 


Sum 


4515.000 







Missing cases 





Valid cases 



712 



39 



Descripti ve Statistics and Chart -2-£ 
Usefulness of CRES ST Presentations 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Plus 


2 


3 


.4 


1.1 


1.1 


Fair Minus 


3 


6 


.8 


2.2 


3.3 


Fair Plus 


4 


27 


3.6 


10.0 


13.4 


Good Minus 


5 


37 


4.9 


13.8 


27.1 


Good Plus 


6 


70 


9.3 


26.0 


53.2 


Excellent Minus 


7 


85 


11.3 


31.6 


84.8 


Excellent Plus 


8 


41 


5.5 


15.2 


100.0 


Omitted 


0 


462 


61.5 


Missing 




Unsure 


9 


1 


.1 


Missing 






99 


19 


2.5 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 

40 | 




Poor Piui Fair Plu» Good Plui EzccUcnt Plus 



Fair Minus Good Minus Excellent Minus 



MecUi 
Mode 
Kurtosis 
S E Skew 
Maximum 


6.171 

7.000 
.052 
.149 

8.000 


Std err 
Std dev 
S E Kurt 
Range 
S\im 


.082 

1.350 

.296 

6.000 

1660.000 


Median 

Variance 

Skewness 

Minimum 


6.000 

1.821 

-.682 

2.000 


Valid cases 


269 


Missing 


cases 482 
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Descriptive Statistics and Chart 27 
Usefulness of CRE SST Technical Reports 



Value Lahel 


Value 


Frequency 


Percent 


Valid 

Percent 


Poor Minus 


1 


1 


.1 


.2 


Poor Plus 


2 


5 


.7 


.9 


Fair Minus 


3 


17 


2.3 


2.9 


Fair Plus 


4 


48 


6.4 


8.3 


Good Minus 


5 


79 


10.5 


13.7 


Good Plus 


6 


156 


20.8 


27.0 


Excellent Minus 


7 


176 


23.4 


30.5 


Excellent Plus 


8 


95 


12.6 


16.5 


Omitted 


0 


162 


21.6 


Missing 




99 


12 


1.6 


Missing 




Total 


751 


100.0 


100.0 



Bar Chart 




Poor Minus Fsir Minns Good Minus EiccUcnt Minus 



Poor Plus Fair Plus 


Good Plus ExotUem 


Plus 






Meeui 


6.199 


Std err 




.057 


Median 


Mode 


7.000 


Std dev 




1.358 


Variance 


Kurt os is 


.299 


S E Kurt 




.203 


Skewness 


S E Skew 


.102 


Range 




7.000 


Minimum 


Maximum 


8.000 


Sum 


3577.000 




Valid cases 


577 


Missing cases 


174 
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18 ^ 



Cum 

Percent 

.2 

1.0 

4.0 

12.3 

26.0 

53.0 

83.5 

100.0 



6.000 

1.844 

-.745 

1.000 



Descrip tive Statistics and Chart 
Usefulness of Journal A rticles /Books Authored bv 

Researchers 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Plus 


2 


7 


.9 


1.3 


1.3 


Fair Minus 


3 


11 


1.5 


2.0 


3.3 


Fair Plus 


4 


31 


4.1 


5.7 


9.0 


Good Minus 


5 


58 


7.7 


10.6 


19.6 


Good Plus 


6 


157 


20.9 


28.7 


48.3 


Excellent Minus 


7 


172 


22.9 


31.4 


79.7 


Excellent Plus 


8 


111 


14.8 


20.3 


100.0 


Omitted 


0 


187 


24.9 


Missing 






99 


17 


2.3 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Poor Plus Fair Plus Good Plus EiccUent Plus 



Fair Minus Good Minus Excellent Minus 77 



Mean 
Mode 
Kurtosis 
S E Skew 
Mcocimum 


6.389 

7.000 
.879 
.104 

8.000 


Std err 
Std dev 
S E Kurt 
Reuige 
Sum 


.056 

1.302 

.209 

6.000 

3495.000 


Median 

Variance 

Skewness 

Minimijm 


7.000 
1.696 
-.920 

2.000 


Valid cases 


547 


Missing 


cases 204 
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Descripti ve Statistics and Chart 29 
Usefuln ess of CRESST Newsletters 











Valid 


Cum 


Value Led>el 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Minus 


1 


4 


.5 


.6 


.6 


Poor Plus 


2 


10 


1.3 


1.5 


2.1 


Fair Minus 


3 


15 


2.0 


2.2 


4.3 


Fair Plus 


4 


60 


8.0 


9.0 


13.3 


Good Minus 


5 


104 


13.8 


15.5 


28.8 


Good Plus 


6 


193 


25.7 


28.8 


57.6 


Excellent Minus 


7 


178 


23.7 


26.6 


84.2 


Excellent Plus 


8 


106 


14.1 


15.8 


100.0 


Omitted 


0 


72 


9.6 


Missing 






99 


9 


1.2 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Poor Phis Fair Plus Good Phis Plus 



Mean 


6.091 


Std err 


.055 


Median 


6.000 


Mode 


6.000 


Std dev 


1.413 


Variance 


1.996 


Kurtosis 


.651 


S E Kurt 


.189 


Skewness 


-.781 


S E Skew 


.094 


Range 


7.000 


Minimum 


1.000 


Maximum 


8.000 


Sum 


4081.000 







Valid cases 670 Missing cases 81 





171 



Descript ive Statistics and Chart 3.P. 
usefulness of CRESST World Wide Web 



Value Label Value 

Poor Minus 1 

Poor Plus 2 

Fair Minus 3 

Fair Plus 4 

Good Minus 3 

Good Plus 6 

Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

99 



Total 







Valid 


Cum 


Frequency 


Percent 


Percent 


Percent 


4 


.5 


.9 


.9 


6 


.8 


1.4 


2.3 


8 


1.1 


1.9 


4.2 


31 


4.1 


7.3 


11.5 


45 


6.0 


10.6 


22.1 


98 


13.0 


23.0 


45.1 


126 


16.8 


29.6 


74.6 


108 


14.4 


25.4 


100.0 


306 


40.7 


Missing 




19 


2.5 


Missing 




751 


100.0 


100.0 





Bar Chart 




Poor Plus Fair Mus Good Phis Excellent Plus 



Mean 
Mode 
Kurtosis 
S E Skew 
Maximum 


6.392 

7.000 
1.268 

.118 

8.000 


Std err 
Std dev 
S E Kurt 
Range 
Sum 


.072 

1.479 

.236 

7.000 

2723.000 


Median 

Variance 

Skewness 

Minimum 


7.000 
2.187 

-1.112 

1.000 


Valid cases 


426 


Missing 


cases 325 
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Descript ive Statistics and Chart 31 
Usefulness of CR ESST Media Products 



Valid Cum 



Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Plus 


2 


5 


.7 


2.3 


2.3 


Fair Minus 


3 


14 


1.9 


6.5 


8.8 


Fair Plus 


4 


15 


2.0 


6.9 


15.7 


Good Minus 


5 


25 


3.3 


11.5 


27.2 


Good Plus 


6 


63 


8.4 


29.0 


56.2 


^cellent Minus 


7 


56 


7.5 


25.8 


82.0 


Excellent Plus 


8 


39 


5.2 


18.0 


100.0 


Omitted 


0 


516 


68.7 


Missing 




Unsure 


9 


1 


.1 


Missing 






99 


17 


2.3 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Fair Minus Good Minus Exceiknt Minus 



Mean 


6.078 


Std err 


.103 


Medi€ui 


6.000 


Mode 


6.000 


Std dev 


1.524 


Variance 


2.323 


Kurtosis 


.070 


S E Kurt 


.329 


Skewness 


-.783 


S E Skew 


.165 


R£uige 


6.000 


Minimum 


2.000 


Maximum 


8.000 


Sum 


1319.000 






Valid cases 


217 


Missing 


cases 534 
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Descript ive Statistics and Chart 2Z 
Usefuln ess of Ideas from CRES^ 



Value Label Value 

Poor Minus 1 
Poor Plus 2 
Fair Minus 3 
Fair Plus 4 
Good Minus ^ 
Good Plus 6 



Excellent Minus 7 

Excellent Plus 8 

Omitted 0 

Unsure 8 

99 

Total 







Valid 


Cum 


Frequency 


Percent 


Percent 


Percent 


2 


.3 


.3 


.3 


5 


.7 


.8 


1.1 


14 


1.9 


2.1 


3.2 


28 


3.7 


4.3 


7.5 


84 


11.2 


12.8 


20.2 


155 


20.6 


23.6 


43.8 


212 


28.2 


32.3 


76.1 


157 


20.9 


23.9 


100.0 


79 


10.5 


Missing 




1 


.1 


Missing 




14 


1.9 


Missing 




751 


100.0 


100.0 





Bar Chart 




Poor Minus 


Fair Minus 


Good Minus ExccUcnt Minus 








Poor Phis Fair 


Plus 


Good Plus ExctUcnt 


Plus 






Mean 


6.478 




Std err 


.051 


Median 


7.000 


Mode 


7.000 




Std dev 


1.319 


Variance 


1.741 


Kurt o sis 


1.140 




S E Kurt 


.190 


Skewness 


-.997 


S E Skew 


.095 




RcUige 


7.000 


Minimum 


1.000 


Maximum 


8.000 




Sum 


4256.000 






Valid cases 


657 




Missing cases 94 








) . 



174 



130 



DescriPt±^e Siiatistics and Chart 33 
Extent CRESST Provided You Useful Information 











Valid 


Cum 


Value Lcd^el 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Minus 


1 


5 


.7 


.7 


.7 


Poor Plus 


2 


12 


1.6 


1.6 


2.3 


Fair Minus 


3 


21 


2.8 


2.8 


5.1 


Fair Plus 


4 


44 


5.9 


6.0 


11.1 


Good Minus 


5 


88 


11.7 


11.9 


23.0 


Good Plus 


6 


209 


27.8 


28.3 


51.3 


Excellent Minus 


7 


226 


30.1 


30.6 


81.9 


Excellent Plus 


8 


134 


17.8 


18.1 


100.0 


Omitted 


0 


5 


.7 


Missing 




Unsure 


9 


1 


.1 


Missing 






99 


6 


.8 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Poor Phis Fair Pius Good Plus K»erllgnt plus 



Mean 


6.246 


Std err 


.052 


Median 


6.000 


Mode 


7.000 


Std dev 


1.420 


Variance 


2.015 


Kurtosis 


1.258 


S E Kurt 


.180 


Skewness 


-1.051 


S E Skew 


.090 


Range 


7,000 


Minimum 


1.000 


Maximum 


8.000 


Sum 


4616.000 






Valid cases 


739 


Missing 


cases 12 
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Descripti ve Stati stics and Chart 34 
Extent CRESST Has Been Useful to Education Conimvn itY 











Valid 


Cvim 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


Poor Minus 


1 


2 


.3 


.3 


.3 


Poor Plus 


2 


6 


.8 


.8 


1.1 


Fair Minus 


3 


12 


1.6 


1.7 


2.8 


Fair Plus 


4 


27 


3.6 


3.8 


6.6 


Good Minus 


5 


75 


10.0 


10.6 


17.2 


Good Plus 


6 


191 


25.4 


27.0 


44.2 


Excellent Minus 


7 


208 


27.7 


29.4 


73.6 


Excellent Plus 


8 


187 


24.9 


26.4 


100.0 


Omitted 


0 


12 


1.6 


Missing 




Unsure 


9 


5 


.7 


Missing 






99 


26 


3.5 


Missing 






Total 


751 


100.0 


100.0 





Bar Chart 




Poor Minus Fair Minus Good Minus Eicdlcnt Minus 



Poor Phis Fair Phis 


Good Phis Excellent 


Plus 






Mean 


6.541 


Std err 


.049 


Median 


7.000 


Mode 


7.000 


Std dev 


1.295 


Variance 


1.677 


Kurtosis 


1.421 


S E Kurt 


.183 


Skewness 


-1.033 


S E Skew 


.092 


Range 


7.000 


Minimum 


1.000 


Mciximum 


8.000 


Sum 


4631.000 






Valid cases 


708 


Missing cases 43 
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Descriptive Statistics an d Chart 35 
What Was Espe cially Useful? 



Value Label 


Value 


Frequency 


Percent 


Valid 

Percent 


Cum 

Percent 


Quality Research 


1 


11 


1.5 


4.8 


4.8 


Newsletters 


2 


22 


2.9 


9.6 


14.5 


Tech Repts 


3 


23 


3.1 


10.1 


24.6 


WWW 


4 


25 


3.3 


11.0 


35.5 


Portfolios 


5 


13 


1.7 


5.7 


41.2 


Tech Issues 


6 


19 


2.5 


8.3 


49.6 


Collaboration 


7 


1 


.1 


.4 


50.0 


Books 


8 


5 


.7 


2.2 


52.2 


Videos 


9 


11 


1.5 


4.8 


57.0 


Topics Covered 


11 


4 


.5 


1.8 


58.8 


CRESST Conference 


12 


1 


.1 


.4 


59.2 


AAIP Database 


13 


6 


.8 


2.6 


61.8 


Per Asst Topics 


14 


21 


2.8 


9.2 


71.1 


Other 


16 


33 


4.4 


14.5 


85.5 


CD ROM 


17 


14 


1.9 


6.1 


91.7 


Multiple Resources 


66 


19 


2.5 


8.3 


100.0 


Never Used 


0 


2 


.3 


Missing 




Omitted 


99 


521 


69.4 


Missing 






Total 


751 


100.0 


100.0 





Chart 35 

What Was Espec ially Useful? 

Bar Chart 




Valid cases 228 Missing cases 523 
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Descript ive Statistics and Cha?it 3.£ 
What Was Problematic? 











Valid 


Cum 


Value Label 


Value 


Frequency 


Percent 


Percent 


Percent 


WWW 


1 


16 


2.1 


28.1 


28.1 


CD ROM 


2 


3 


.4 


5.3 


33.3 


Relevance Missing 


3 


6 


.8 


10.5 


43.9 


Videos 


4 


6 


.8 


10.5 


54.4 


Dated Information 


5 


3 


.4 


5.3 


59.6 


Other 


6 


23 


3.1 


40.4 


100.0 


Never Used 


0 


12 


1.6 


Missing 




Omitted 


99 


682 


90.8 


Missing 






Total 


751 


100.0 


100.0 





Chart 36 

What Was Problematic? 



Bar Chart 
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Descriptive Stat istics and Chart 37. 
What New Reseach or Product is Neede d? 



Value Label Value 

Scoring 1 
More Tests 3 
More Riibrics 4 
Technical Issues 5 
Other 6 
Standards Info 7 
More of the same 8 
Info about products 9 
Portfolios 10 
Multiple 66 
Never Used 0 
Omitted 99 



Total 







Valid 


Ci]m 


Frequency 


Percent 


Percent 


Percent 


4 


.5 


2.0 


2.0 


23 


3.1 


11.6 


13.6 


2 


.3 


1.0 


14.6 


3 


.4 


1.5 


16.1 


125 


16.6 


62.8 


78.9 


6 


.8 


3.0 


81.9 


27 


3.6 


13.6 


95.5 


4 


.5 


2.0 


97.5 


3 


.4 


1.5 


99.0 


2 


.3 


1.0 


100.0 


1 


.1 


Missing 




551 


73.4 


Missing 




751 


100.0 


100.0 





Chart 37 

What New Reseach or Product is Needed 




APPENDIX E 

Normality Plots and Nonparametric Tests 
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QQ-Plots from SPSS were plotted for the Descriptive Survey data in order to 
determine if the distributions could be analyzed with parametric tests, such as analysis of 
variance. According to Nomsis (1994) QQ Plots that form a relatively straight line 
indicate that the data may be considered a normal distribution (Nomsis, 1994). She 
goes on to say in a different section: “It is almost impossible to find data that are exactly 
normally distributed. For most statistical tests, it is sufficient that the data are 
approximately normaUy distributed.” 

Figure 1 

Normal q-q plot for Quality of CRESST Products 



Normal Q-Q Plot of Quality of Products 




Observed Value 



Expected Nomnl Vahia Expected HonxMl Vibe 



Figure 2 

Normal q-q plot for Quality of Tech Reports 



Normal Q-Q Plot of Quality of Tech Reports 




Observed Value 



Figures 

Nomud q-q plot for Quality of Presentations 



Normal Q-Q Plot of Quality of Presentations 




Observed Value 



o 
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tdHosnl VahM BsptettdHosial Vahn 



Figures 

Normal q-q plot for Quality of Topics Covered 



Normal Q-Q Plot of Quality of Topics Covered 




Observed Value 



Figure 6 

Normal q-q plot for Useful to Education Community 



Normal Q-Q Plot of Useful to education community 




Observed Value 



er|c 
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Figure 3 

Normal q-q plot for Extent Provided You Useful Information 



Normal Q>Q Plot of Extent provided you useful infor. 




Observed Value 



Conclusion. These plots and those for all the dependent variables of the 
descriptive survey approach a straight line. This lends support to our ability to safely 
conduct parametric tests, primarily Analysis of Variance (ANOVA), to detect significant 
differences between dependent and independent variables. 

Nonparametric Tests 

Various nonparametric chi-square tests including Chi-square Pearson, 
Likelihood Ratio, Phi Coefficient, the Coefficiency of Contingency, and Cramer’s V, 
were run to detect associations between dependent and independent variables of the 
CRESST Descriptive Survey. Eta, for nonparametric distributions, which provides a 
measure of strength between the variables, was also calculated. A sample of the results 
are provided in Table 1 focusing on the independent variable “sharing CRESST research 
with others” and the dependent variable of “overall quality of CRESST research.” 
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Table 1 



Nonparametric Measurements of Association 
CRESST Descriptive Questionnaire 
Quality of CRESST R&D by Shared CRESST with Others 



DV 

QualRD 



rV Test Cases 

shared w/o Pearson 684 

Ukliehood 684 

Phi 684 

Cramer’s 684 

Contingen 684 

Pearson R 694 

Spearman 694 

Eta q/dep 694 

Eta s/dep 694 



Value 


df 


Signif. 


83.76 


18 


.0000 


80.22 


18 


.0000 


.35 


n/a 


.0000 


.20 


n/a 


.0000 


.33 


n/a 


.0000 


.28 


n/a 


.0000 


.29 


n/a 


.0000 


.29 


n/a 


n/a 


.30 


n/a 


n/a 



In the case presented, we can reject the hypothesis that there is no association 
between the ratings of the overall quality of CRESST R&D and usefulness to the 
education community across the independent variable sharing CRESST research, just as 
we did in our analysis of variance in the body of this dissertation. This is based on the 
P-values for the Pearson chi square, phi coefficient, likelihood ratio, the coefficient of 
contingency, and Cramer’s V, all less than .05. Eta { 29 ) suggests a fairly strong 
relationship. 

A further analysis showed that we could reject the nuU hypothesis of no 
association between all dependent variables and the independent variable “Sharing 



CRESST research with others” except for the usefulness of CRESST media products 
P=.09 and usefulness of CRESST presentations P=.07. This almost exactly matches 
our ANOVA findings. ANOVA showed a significant difference between the dependent 
variable “usefulness of CRESST media products” and “sharing CRESST research.” 
However, the Bonferoni and Scheffe tests showed significance only across two groups, 
group 1 , shared CRESST research once and group 3, shared CRESST research 4 times 
or more. The results suggest that we are quite safe in using ANOVA to test the data 
from the CRESST Descriptive Questionnaire. 

Results presented in Table 2 from nonparametric measurements of association, 
also confirm our ANOVA findings in the main body of this dissertation, that a significant 
relationship between the dependent variable “overall quality of CRESST research” and the 
independent variable, group audiences, does not exist. 
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Table 2 



DV 

QualRD 



er|c 



Nonparametric Measurements of Association 
CRESST Descriptive Questionnaire 
Quality of CRESST R&D by Group Audiences 



IV 


Test 


Cases 


groups 


Chi- 


694 




Pearson 






Likliehood 


694 




Phi 


694 




Cramer’s 


694 




Contingen 


694 




Pearson R 


694 




Spearman 


694 




Etaq/dep 


694 




Etag/dep 


694 



Value 


df 


Signif. 


21.07 


18 


.27561 


20.67 


18 


.29639 


.1742 


n/a 


.27561 


.10061 


n/a 


.27561 


.17168 


n/a 


.27561 


-.02081 


n/a 


.58409 


-.00228 


n/a 


.95211 


.08896 


n/a 


n/a 


.11214 


n/a 


n/a 
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